Principal Cloud Engineer

About Graphcore At Graphcore, we’re building the future of AI compute.We’re a team of semiconductor, software and AI experts, with deep experience in creating the complete AI compute stack - from silicon and software to infrastructure at datacenter scale. As part of the SoftBank Group, backed by significant long-term investment, we are delivering key technology into the fast-growing SoftBank AI ecosystem.To meet the vast and exciting AI opportunity, Graphcore is expanding its teams around the world.We are bringing together the brightest minds to solve the toughest problems, in a place where everyone has the opportunity to make an impact on the company, our products and the future of artificial intelligence. Job Summary We are looking for an experienced Principal Engineer to join our Cloud Platform Team and help develop and deploy clouds and services. Working closely with our colleagues in Software Platform, Datacentre Operations and Product Development teams, you will deploy services on our fleet of cutting-edge AI systems. As part of our Software Platform organisation, you will be involved in the cloud integration, validation, performance benchmarking, optimisation, and development of our high-performance AI solutions. These include in-house AI systems alongside off-the-shelf high-performance servers, switches and storage solutions. This is a hand-on technical role requiring a solid background in the use of cloud infrastructure, deployment using Infrastructure-as-Code, observability, high-performance networking and storage systems. You may have been working in an IT organisation, a datacentre, a cloud provider or as a developer of orchestration or cloud services. The Software Platform team at Graphcore We build Graphcore products into large-scale AI solutions for our customers and the Cloud Platform Team is responsible for providing such systems to both internal users via private clouds and customers via our own public clouds. Often the internal systems will be using and developing pre-release hardware and software, so it’s vital you are comfortable with unproven components. Responsibilities and Duties Operate and extend existing OpenStack-based cloud services and contribute to the deployment and development of new ones. You will be responsible for significant technical initiatives and projects, mentoring a small team of more junior engineers in best engineering practices. Develop and operate end-user services on our clouds and support internal users in their use. You will turn end-user and product requirements into deployed services. Help to build automation to collect and analyse metrics and other observability data from the cloud services to support clear identification and reporting of any issues. Work with users to provide information of any product-related issues to Engineering and QA departments. Work with our Datacentre Operations Engineers to maintain and operate the fleet of AI systems at peak performance in our private clouds. Configure and test new Graphcore AI hardware and systems using Continuous Deployment and Infrastructure-as-code in internal and external datacentres. Drive corrective actions for systems that are not operating correctly, working with DC operations and Graphcore Engineering as required. Work with external vendors of off-the-shelf switches, servers and storage solutions to specify, benchmark and integrate 3rd party products into our Cloud Reference Design. Skills and Experience [ALL REQUIRED] Bachelor's degree or equivalent practical experience in a relevant subject. Solid infrastructure or IT experience with a proven track record of delivering technical output as an individual contributor. Experience managing or operating on-premises or private-cloud environments. Experience specifying, scoping, estimating and detailing work plans in an AGILE and SCRUM framework, including priorities, risks, issues, impacts and constraints. Expert-level, proven Linux scripting ability (bash and python required). Expert-level, proven Linux system administration (Ubuntu, RHEL and variants). Experience with a version control system (preferably Git) and using it to manage system configuration or automation. Experience with Continuous Integration or testing pipelines using GitLab, GitHub or similar. Hands-on experience deploying services into public or private clouds using Infrastructure-as-Code (IAC). A solid understanding of the technologies underpinning cloud services (APIs, virtualisation of CPUs, IO, systems), virtual networks, block storage, resource management and monitoring. Expert with IAC automation tools (e.g. Terraform/OpenTofu, Ansible, Packer). Experience with container deployment and management tools (e.g. docker, podman, apptainer). Experience with solutions for monitoring and observability. e.g. Grafana, Prometheus, OpenSearch/ElasticSearch, Loki, Mimir, OpenTelemetry, Fluentd ,Kafka Excellent communication and presentation skills, and experience dealing with end-users of IT or cloud services. An ability to work independently and lead others on critical infrastructure without oversight, and with a focus on end-user availability.    Desirable but not required: Experience with OpenStack deployments or the technologies they rely on (e.g. Ceph, Open vSwitch, KVM, QEMU ). Experience with High Performance Computing (HPC) environments using SLURM or similar batch workload solutions. Strong skillset and experience in end-to-end deployment automation and CI of containerised services. Complete automation of pipelines for build, test, deploy, manage, alert, destroy, rebuild. Experience with managing production Kubernetes clusters and workloads. Experience with workload queue management systems (SLURM, LSF, Kueue). Experience with managed switch configuration (e.g. EOS, SONiC, DNOS). Programming experience with Python3 utilising classes and inheritance. Programming experience with Go. Benefits In addition to a competitive salary, Graphcore offers flexible working, a generous annual leave policy, private medical insurance and health cash plan, a dental plan, pension (matched up to 5%), life assurance and income protection. We have a generous parental leave policy and an employee assistance programme (which includes health, mental wellbeing, and bereavement support). We offer a range of healthy food and snacks at our central Bristol office and have our own barista bar! We welcome people of different backgrounds and experiences; we’re committed to building an inclusive work environment that makes Graphcore a great home for everyone. We offer an equal opportunity process and understand that there are visible and invisible differences in all of us. We can provide a flexible approach to interview and encourage you to chat to us if you require any reasonable adjustments. Sponsorship Applicants for this position must hold the right to work in the UK. Unfortunately at this time, we are unable to provide visa sponsorship or support for visa applications.