Sr Principal / Architect Engineer, Software - Backend, AI Observability/Evaluat

Our Mission At Palo Alto Networks®, we’re united by a shared mission—to protect our digital way of life. We thrive at the intersection of innovation and impact, solving real-world problems with cutting-edge te

Our Mission

At Palo Alto Networks®, we’re united by a shared mission—to protect our digital way of life. We thrive at the intersection of innovation and impact, solving real-world problems with cutting-edge technology and bold thinking. Here, everyone has a voice, and every idea counts.

If you’re ready to do the most meaningful work of your career alongside people who are just as passionate as you are, you’re in the right place.

Who We Are

In order to be the cybersecurity partner of choice, we must trailblaze the path and shape the future of our industry. This is something our employees work at each day and is defined by our values: Disruption, Collaboration, Execution, Integrity, and Inclusion.

We weave AI into the fabric of everything we do and use it to augment the impact every individual can have. If you are passionate about solving real-world problems and ideating beside the best and the brightest, we invite you to join us!

This role is remote, but distance is no barrier to impact. Our hybrid teams collaborate across geographies to solve big problems, stay close to our customers, and grow together. You will be part of a culture that values trust, accountability, and shared success where your work truly matters.

About us

Chronosphere, a Palo Alto Networks company, is the observability platform built for control in the modern, containerized world. Chronosphere empowers customers to focus on the data and insights that matter by reducing complexity, optimizing costs, and remediating issues faster.

Chronosphere reduces data volumes and associated costs by 84% on average while saving developers thousands of hours. Recognized as a leader by major analyst firms, Chronosphere is trusted by the world's most innovative brands, including DoorDash, Affirm, and Zillow.

About our roles

We are building AI Observability and Evaluation capabilities on Chronosphere’s observability platform to help developers understand and improve the accuracy, relevance, performance, and safety of AI models, agents, and retrieval-augmented generation (RAG) applications.

Our team enables customers to systematically test, measure, and benchmark AI agents and applications before deployment and monitor how they behave in production. We turn complex telemetry and evaluation data into insights that help developers build and operate reliable AI agents and applications at scale.

We are looking for experienced engineers with a strong product sense who asks “why” as often as “how”. You’ll partner with product managers, engineers, and customers to translate emerging AI observability and evaluation needs into scalable backend systems and cohesive developer workflows.

You’ll help shape the technical roadmap and architecture, taking features from early exploration through production delivery.

You will

• Build backend capabilities that help developers observe and evaluate AI models, agents, and RAG applications before and during production

• Design systems for systematic testing, measurement, and benchmarking of AI agents and applications accuracy, relevance, performance, and safety

• Build scalable APIs and data processing systems that contextualize AI telemetry and evaluation results, helping customers understand application behavior and identify opportunities for improvement

• Help shape the technical roadmap and architecture for AI Observability and Evaluation, building on Chronosphere’s rapidly scaling platform

• Collaborate with product managers, deeply technical teammates, and customers to scope, validate, and deliver integrated features and developer workflows

• Improve our own system observability (we use our own product heavily)

• Improve the performance, reliability, and cost efficiency of the distributed systems powering AI Observability and Evaluation

• Mentor engineers and help maintain our engineering standards

You have

• 12-15+ years of experience in software engineering

• Strong experience in one or