Overview

As a Principal Software Engineer in the Artificial Intelligence group, you will play a crucial role in building and optimizing the core software infrastructure that powers AI-driven solutions. You will focus on architecting and deploying highly scalable, production-ready backend systems that support AI assistants, intelligent agents, and foundational AI services. Collaborating with machine learning engineers and cross-functional teams, you will drive best practices in software engineering, DevOps, Kubernetes-based deployments, and backend service development. Your expertise will be instrumental in accelerating AI innovation by ensuring robust, reliable, and efficient system operations.

Responsibilities

Design and implement high-performance backend architectures that seamlessly integrate with AI-powered products. Focus on building modular, fault-tolerant, and efficient services that support large-scale AI workloads while ensuring low-latency interactions between data pipelines, inference engines, and enterprise applications.
Develop robust model-serving APIs and containerized microservices that enable real-time AI inference and batch processing with high throughput and low latency.
Implement end-to-end monitoring, logging, and alerting solutions to ensure AI systems operate reliably at scale.
Improve scalability by designing distributed systems that efficiently handle AI workloads and inference pipelines.

Own Kubernetes-based deployments by developing and maintaining Helm charts, Kubernetes operators, and cloud-native workflows to streamline AI model deployment.
Automate infrastructure management using Infrastructure as Code (IaC) tools like Terraform or CloudFormation.
Optimize CI/CD pipelines for AI applications, ensuring smooth model retraining, testing, and deployment cycles.
Improve security and compliance by implementing best practices in access control, container security, and vulnerability management.

Partner closely with AI/ML teams to ensure seamless model integration into production environments.
Lead architecture discussions and provide strategic technical guidance on AI platform evolution.
Mentor and guide engineers to enhance team skills in backend development, DevOps, and cloud technologies.

Requirements

Strong backend development experience in Python (preferred) or Java, with expertise in building RESTful APIs, microservices, and event-driven architectures.
Deep understanding of Kubernetes and container orchestration, with experience in deploying AI/ML workloads at scale.
Expertise in DevOps and CI/CD pipelines, including experience with Jenkins, GitHub Actions, ArgoCD, or similar tools.
Cloud expertise (AWS/GCP/Azure), including hands-on experience with cloud-native services for AI workloads (e.g., S3, Lambda, EKS/GKE/AKS, DynamoDB, RDS etc.).
Experience in performance tuning and system optimization for large-scale AI/ML workloads.

Proven ability to collaborate with ML engineers, data scientists, data engineers and product teams to deliver AI-powered solutions efficiently.
Experience in technical leadership, driving architectural decisions, and mentoring engineers.
Strong problem-solving skills, with the ability to balance trade-offs between scalability, maintainability, and performance.

Preferred Experience

Prior experience working with AI/ML pipelines, model serving frameworks, or distributed AI workloads.
Experience in AI observability, monitoring model drift, and optimizing inference latency.
Understanding of cybersecurity, observability, or related domains to enhance AI-driven decision-making.

Splunk, a Cisco company, is an Equal Opportunity Employer and all qualified applicants will receive consideration for employment without regard to race, color, religion, gender, sexual orientation, national origin, genetic information, age, disability, veteran status, or any other legally protected basis.

Note:

Apply Now

Explore open roles >

Splunk's Hiring Practices

^{Splunk turns machine data into answers. Organizations use market-leading Splunk solutions with machine learning to solve their toughest IT, Internet of Things and security challenges.}

^{We value diversity, equity, and inclusion at Splunk and are committed to equal employment opportunity. Qualified applicants receive consideration for employment without regard to race, religion, color, national origin, ancestry, sex, gender, gender identity, gender expression, sexual orientation, marital status, age, physical or mental disability or medical condition, genetic information, veteran status, or any other consideration made unlawful by federal, state or local laws. We consider qualified applicants with criminal histories, consistent with legal requirements. Click here to review the US Department of Labor’s EEO is The Law notice. Please click here to review Splunk’s Affirmative Action Policy Statement. If you need assistance or an accommodation to apply or during the hiring process, please let us know by completing our Accommodation Request form.}

^{Splunk also has policies in place to protect the personal information candidates disclose to us as part of the application process. Please click here to review Splunk’s Career Site Privacy Policy.

Splunk does not discriminate against employees or applicants because they have inquired about, discussed, or disclosed their own pay or the pay of another employee or applicant. Please click here to review Splunk’s Pay Transparency Nondiscrimination Provision.

Splunk is committed to the health and safety of our employees and customers. Splunk is impacted by the mandates outlined for U.S. Government contractors in President Biden’s Path out of the Pandemic: COVID-19 Action Plan. As a result, Splunk requires U.S. employees, whether assigned to an office or 100% remote, to provide proof of full vaccination, as defined by the CDC. Splunk provides reasonable accommodations for employees who have qualifying medical or religious reasons.

Splunk is also committed to providing access to all individuals who are seeking information from our website. Any individual using assistive technology (such as a screen reader, Braille reader, etc.) who experiences difficulty accessing information on any part of Splunk’s website should send comments to accessiblecareers@splunk.com. Please include the nature of the accessibility problem and your e-mail or contact address. If the accessibility problem involves a particular page, the message should include the URL of that page.

Splunk doesn't accept unsolicited agency resumes and won't pay fees to any third-party agency or firm that doesn't have a signed agreement with Splunk.

To check on your application click here.}