Resident Engineer

ROLE SUMMARY: The Resident Incident Engineer is responsible for maintaining the availability, performance, resilience, and operational readiness of DDN storage infrastructure deployed within a customer / NVIDIA Cloud Partner (NCP) environment. The role will initially support a mid-tier NCP deployment that is expected to grow into one of the largest storage environments in the region. The engineer will act as a trusted on-site technical resource, working closely with the customer’s infrastructure, platform, Kubernetes, networking, security, and operations teams. The role requires hands-on expertise across DDN storage platforms, EXAScaler/Lustre, Linux, high-speed networking, data-centre infrastructure, Kubernetes, and the Container Storage Interface (CSI). The engineer will provide proactive operational support, incident response, problem management, capacity planning, performance optimization, and technical coordination with DDN Support and Engineering. KEY RESPONSIBILITIES • Provide on-site operational ownership for DDN storage systems and associated EXAScaler/Lustre environments. • Monitor storage health, availability, capacity, performance, alerts, and system trends. • Respond to incidents, service requests, and operational events in accordance with agreed severity and response targets. • Lead technical triage, fault isolation, recovery, and root-cause analysis for storage, file-system, Linux, networking, and integration issues. • Coordinate with customer operations teams, DDN Support, Engineering, hardware vendors, and other stakeholders during major incidents. • Maintain and troubleshoot EXAScaler/Lustre file systems, metadata services, storage targets, client connectivity, failover, and recovery procedures. • Support Kubernetes environments that consume DDN storage, including cluster integration, persistent volumes, storage classes, access modes, and workloadlevel storage troubleshooting. • Configure, validate, and troubleshoot CSI drivers and related Kubernetes storage components Investigate issues across the Kubernetes-to-storage path, including pods, nodes, kubelet, CSI controller and node plugins, APIs, mounts, permissions, networking, and backend storage. • Support NFS, parallel file-system, block, and other storage access protocols where applicable. • Perform planned maintenance, upgrades, expansions, migrations, hardware replacements, and configuration changes using approved change-management processes. • Develop and maintain operational runbooks, standard operating procedures, recovery procedures, escalation guides, and troubleshooting documentation. • Conduct regular health checks, capacity reviews, performance assessments, and operational-readiness reviews. • Identify risks, recurring failure patterns, single points of failure, and opportunities to improve reliability and automation. • Validate backup, restore, replication, disaster-recovery, and business-continuity procedures where applicable. • Support capacity forecasting for storage, metadata, network, client, and datagrowth requirements. • Assist with performance benchmarking, workload analysis, tuning, and servicelevel reporting. • Maintain accurate configuration management, asset records, incident records, problem records, and installed-base information. • Participate in on-call, out-of-hours, or escalation coverage as required by the customer environment. • Provide technical briefings, knowledge transfer, and operational training to customer teams. • Contribute to implementation schedules, risk and mitigation plans, acceptance testing, handover, and customer sign-off. Required Qualifications and Experience • Bachelor’s degree or equivalent experience in Computer Science, Engineering, Information Technology, or a related discipline. • At least 5 years of experience in infrastructure engineering, storage administration, systems administration, data-centre operations, or a related field. • Hands-on experience supporting enterprise storage, HPC, AI, cloud, or highperformance computing environments. • Strong Linux systems administration and troubleshooting skills. +1.800.837.2298 · [email protected] · ddn.com http://ddn.com • Experience with DDN storage platforms and/or EXAScaler/Lustre environments. • Experience with file systems such as Lustre, NFS, GPFS, or other parallel or distributed file systems. • Working knowledge of Kubernetes architecture and day-to-day cluster operations. • Working knowledge of the Container Storage Interface, including CSI controllers, node plugins, volume lifecycle operations, mounts, snapshots, expansion, and topology considerations. • Experience troubleshooting persistent storage consumed by Kubernetes workloads. • Strong understanding of IP networking, Ethernet, InfiniBand, or other high-speed interconnects. • Ability to interpret system logs, Kubernetes events, CSI logs, metrics, alerts, and performance data. • Experience with monitoring, observability, and alerting platforms. • Experience with incident management, problem management, change management, and technical escalation processes. • Ability to troubleshoot methodically across hardware, operating systems, storage, file systems, Kubernetes, networking, and application layers. • Strong understanding of data-centre fundamentals, including power, cooling, rack layouts, cabling, remote management, and hardware replacement procedures. • Strong written and verbal communication skills. • Ability to work independently on a customer site while collaborating effectively with distributed technical teams. • Willingness and ability to participate in on-call or after-hours support when required. DESIRABLE SKILLS: • Experience supporting large-scale or rapidly growing storage environments. • Experience with Kubernetes distributions such as Red Hat OpenShift, SUSE Rancher, VMware Tanzu, or equivalent platforms. • Familiarity with Helm, Operators, Custom Resource Definitions, YAML, and Kubernetes troubleshooting tools. • Experience with Prometheus, Grafana, Alertmanager, OpenTelemetry, or comparable observability platforms. • Working knowledge of Ansible, Python, Bash, or other automation technologies. • Experience with Git-based configuration management and infrastructure-ascode practices. +1.800.837.2298 · [email protected] · ddn.com http://ddn.com • Familiarity with S3 or object-storage concepts and data-movement tools. • Experience with authentication and authorization technologies such as LDAP, Active Directory, Kerberos, and TLS certificates. • Knowledge of DNS, NTP, DHCP, routing, VLANs, MTU, firewall policies, and network troubleshooting. • Experience with high-availability design, quorum, failover, replication, and disaster recovery. • Familiarity with ITIL-based service management processes. • Experience preparing operational reports, service reviews, capacity forecasts, and executive summaries. • Relevant certifications in Linux, Kubernetes, cloud, networking, or storage. • Knowledge of Mandarin Chinese would be an advantage