Are you interested in working with the World’s leading AI-powered Quality Engineering Company? Ready to advance your career, team up with global thought leaders across industries and make a difference every day? Join us at Qualitest!
We are looking for a Sr. Data Scientist in Test to join our growing team in United States!
Role - Sr. Data Scientist in Test
Location - Remote (US)
Job Description:
Develop test strategies for evaluating AI/ML models, ensuring outputs align with business requirements and user expectations.
Design and execute evaluation pipelines using frameworks like DeepEval for generative AI model testing.
Automate the evaluation of model accuracy, fluency, factuality, safety, and bias.
Strong background in AI/ML testing, with hands-on experience in generative AI or NLP models.Proficiency in Python and testing libraries like DeepEval, Langsmith, LangTest, or Hugging Face evaluation tools.
Knowledge of AI-specific testing metrics and adversarial testing methodologies.
Use metrics like BLEU, ROUGE, perplexity, and embedding similarity to evaluate generative model output.
Role Overview:
The Data Scientist in Test will be responsible for designing, developing, and implementing testing strategies for AI-powered applications, such as chatbots and generative AI models. The candidate will evaluate model quality, performance, robustness, and ethical considerations using state-of-the-art testing frameworks.
Roles and Responsibilities:
• Develop test strategies for evaluating AI/ML models, ensuring outputs align with business requirements and user expectations.
• Design and execute evaluation pipelines using frameworks like DeepEval for generative AI model testing.
• Automate the evaluation of model accuracy, fluency, factuality, safety, and bias.
• Create adversarial test cases to validate AI behavior under edge scenarios, such as data poisoning, jailbreaks, and prompt injections.
• Assess and validate the retrieval augmented generation (RAG) system, including retrieval accuracy and latency.
• Build automated testing for conversational AI chatbots, covering dialog coherence, context retention, and response diversity.
• Use metrics like BLEU, ROUGE, perplexity, and embedding similarity to evaluate generative model output.
• Collaborate with data scientists, ML engineers, and developers to address bugs and performance issues.
• Implement tools for test data generation that simulate real-world user inputs and edge cases.
• Report model performance through dashboards and metrics-driven reporting frameworks.
Skills and Qualifications:
• Strong background in AI/ML testing, with hands-on experience in generative AI or NLP models.
• Proficiency in Python and testing libraries like DeepEval, Langsmith, LangTest, or Hugging Face evaluation tools.
• Knowledge of AI-specific testing metrics and adversarial testing methodologies.
• Experience with model evaluation frameworks like MLflow, Weights & Biases, or custom pipelines.
• Familiarity with LLM architectures (e.g., GPT, BERT) and concepts like prompt engineering and RAG.
• Strong analytical mindset and problem-solving skills for identifying AI model failures.• Experience with tools like Jupyter Notebooks, Pandas, NumPy, and visualization libraries (e.g., Matplotlib).
Why QualiTest?
Be a part of a company who strives to support for diversity and inclusion in the workplace – we are one, we are many at Qualitest. Celebrate culture, share knowledge with engineers from around the globe, and inspire each other through our differences. Local and global opportunities – we offer you internal rotation and international mobility opportunities to grow your career. Clear view of your career and progression with the company – Qualitest is growing massively and giving you the opportunity to grow with us. Work hard and play harder with our flexible and casual culture. Take a break from work and join an employee event, or enjoy the amenities and games provided from one of our Employees Centers. Save your earnings and prepare for your future by enrolling in our 401k plan where Qualitest will match your contributions accelerating your savings plan. Take care of health with enrollment into one of our competitive healthcare benefits. Qualitest will match towards your HSA if you choose to participate. Never stop experimenting and learning with Qualitest Tech academy: 3000+ training courses, mentorship programs, technical tribes, sponsored certifications, leadership programs and much more Stay active and get rewarded with our Corporate Wellness Program. We pay your Gym membership and giving you opportunities to Earn additional vacation times for attendance the gym! Competitive Salary of $145-155k per year Earn bonuses via our Client Referral and Employee Referral Program’s. Refer and earn – tap your network for net-worth. Planning a vacation? Looking for car insurance? Get access to Qualitest Employee Perks for discounts on anything from travel to electronics. With so many offerings the savings are endless!Intrigued to find more about us?
Visit our website at Careers - Qualitest GroupIf you like what you have read, send us your resume and let’s start talking!