KubeCon + CloudNativeCon China 2025
June 10–11, 2025 · Hong Kong.
KubeCon + CloudNativeCon China 2025 brought adopters and technologists from the open source and cloud native communities to Hong Kong for two days of keynotes, breakouts, lightning talks and the Solutions Showcase.
Sessions from end user organizations
The event published no end user track, so the sessions below are those whose speakers listed an organization that meets the CNCF End User definition as their company in the official schedule: an organization that uses cloud native technologies internally rather than selling or supporting them. They include keynotes from 5miles, Bilibili, eBay, HKGAI and iFlytek alongside talks from CNCF End User members such as Ant Group, Bloomberg and eBay, and from adopters such as ByteDance, China Mobile, Horizon Robotics, KPMG Australia, Kuaishou, PayPal, vivo and Xiaohongshu, plus educators from the Hong Kong Institute of Information Technology.
21 sessions, presented by speakers from organizations that meet the CNCF End User definition. All times are HKT (UTC+8), local to Hong Kong. Some sessions overlap; talks in the same time slot run concurrently.
Source: official Sched listing · Verified November 2, 2026. This conference has ended; the schedule below is the final record. See the official schedule for the full program. Xuzheng Chang, Dong Jiang Training massive AI models at scale is tough—but doing it efficiently in Kubernetes is even tougher. In this keynote, we’ll share how iFlytek tackled key challenges in large-scale model training, including low GPU utilization, fragile workflows, and resource contention across teams. By leveraging Volcano, they boosted GPU usage by over 40%, and cut failure recovery time by 70%. This talk offers a quick but powerful look at how intelligent scheduling and orchestration can unlock performance, reliability, and fairness in multi-tenant AI platforms. Level 16 | Grand Ballroom I Roby Chen, Aaron Xu The release of HKGAI V1 marks a new chapter in the development of AI in Hong Kong. Leveraging the strengths of both local connectivity and global outreach, the HKGAI team embraces the open-source community to tackle challenges ranging from optimizing high-performance computing clusters to exploring cutting-edge AI models. Looking ahead, Hong Kong aims to further integrate resources from mainland China and the international community, deepening technological innovation and application expansion, and contributing a "Hong Kong Solution" to global AI standards and use cases. Level 16 | Grand Ballroom I Tao Peng, Xiaoya Xia, Wenbo Qi, Han Jiang, Wenpeng Li As the demand for scalable AI/ML grows, efficiently distributing AI models in cloud-native infrastructure has become a pivotal challenge for enterprises. The panel dives into the technical and operational strategies for deploying models at scale -- from optimizing model storage and transfer to ensuring consistency across clusters and regions. Experts from different companies and CNCF projects will debate critical questions like: How can Kubernetes-native workflows automate and accelerate model distribution while minimizing latency and bandwidth costs? How to efficiently distribute huge models sizing hundreds of GBs or TBs? What are the challenges proposed by distributed inference and the prefilling-decoding architecture? How are models updated in the reinforcement learning post-training paradigm? What role do standards like OCI artifacts or specialized registries play in streamlining versioned model delivery? Level 19 | Crystal Court I Yixiang Chen, Yingcai Xue For an event-driven distributed system like Kubernetes, where components communicate by synchronizing incremental data through the KubeAPIServer, the metadata system is the most critical component. The ETCD is the only official supported metadata system, but some projects like kine explored alternative metadata storage, but they're either not open-sourced or have performance issues.
This talk covers ByteDance's work on high-performance Kubernetes metadata systems. It summarizes ETCD's production issues, analyzes Kubernetes' metadata storage requirements and introduces how we solve it with kubebrain.
Actual results from large-scale environments (over 20K nodes, 1M pods over years) show that KubeBrain enhances cluster performance and stability.
This talk helps understand the challenges of metadata systems in large-scale clusters and provides insights into an open-source solution that has been practiced in ByteDance's production environment. Level 19 | Crystal Court II Tao Peng, Chenyu Zhang, Fog Dong, Xudong Wang AI has become a prominent figure in the cloud native ecosystem and there continues to be massive adoption in this emerging field. As frameworks and approaches are introduced, a pattern has emerged which threatens the ability to manage at scale: each implementation introduces their own format, runtime, and different ways of working, fragmenting the ecosystem. On other hand, open standards are the backbone of cohesive and scalable ecosystems. This panel discussion seeks to explore the importance of defining standards within the CNCF ecosystem, particularly focusing on AI/ML artifacts. Beyond the advantages of the standard in facilitating integration with existing cloud native tools, this conversation will delve into how the standards can serve as a foundation for innovation. Join us to understand how standardization with innovative approaches can advance the cloud native AI landscape. Level 19 | Crystal Court I Zhihao Xu, Chen Yangxue End-to-end large model training is crucial for advancing autonomous driving technology. Horizon Robotics leads in this field by leveraging deep learning algorithms and chip design. They efficiently train and deploy advanced perception models like Sparse4D using cloud-native technologies.
Training these models poses challenges, such as managing massive video data and numerous small files. Ensuring high-performance training with over 2000 GPUs on RDMA, quickly identifying different failures, and diagnosing issues in large-scale training.
This session covers how Horizon Robotics manages large-scale training on Kubernetes. It highlights the role of distributed data caching, network topology awareness, and job affinity scheduling in optimizing a 2000 GPU training job. We'll also discuss strategies for restoring interrupted training jobs through backup machine replacement to enhance task resilience. Furthermore, experiences with CNCF projects like Volcano, Fluid, and NPD will be shared. Level 19 | Crystal Court I Di Xu, Xu Liu With the growing adoption of Kubernetes, managing configurations and ensuring compliance across extensive clusters becomes increasingly complex. Kyverno, a native Kubernetes policy engine, offers a streamlined solution to these challenges. In this session, we'll explore how adopting Kyverno can enhance efficiency, simplify operations, centralize control, and reduce maintenance in Kubernetes environments. We'll demonstrate how Kyverno empowers organizations to effectively manage policies and tackle the unique challenges of large-scale Kubernetes deployments. Drawing from real-world experiences, we will share valuable lessons and best practices that facilitate seamless policy integration and management. Attendees will gain practical insights and tools to optimize their Kubernetes environments using Kyverno. Level 19 | Crystal Court II Liyi Huang, Kaixi Fan Join us as we celebrate nearly a decade of Cilium, now the de-facto standard CNI for Kubernetes and a cornerstone of cloud native networking, observability, and security. This session provides updates on the latest Cilium release and showcases how its unified eBPF-powered stack is transforming Kubernetes environments by replacing fragmented toolchains with seamless, secure, scalable, and simplified solutions. Hear about how Cilium is simplifying the cloud native stack and solidifying its role as the comprehensive networking and security solution for modern cloud native architectures from contributors and end users Bytedance and Isovalent. Level 21 | Pearl Pavilion Pei (Sandy) Wang Tech startup in early stage normally aim low running cost on infrastructure spend but fast development and delivery. When there are a first few clients onboard, disaster recovery plan is a must have. When DR is required and an agreed RTO is 6 hours for example, how to not only remain low running cost but also to meet agreed RTO and SLA, our DR plan and implementation is a success to share with the audience. We onboarded container orchestration platform Kubernetes, DevOps best practices, for example Infrastructure-and-Configuration-as-Code and Pipeline-as-Code. Our DR implementation only spends a minimum cost on always-on resources. When a DR incident happens, automated pipelines will bring up on-demand resources that include a Kubernetes cluster, and geo-recover database and storage, then deploy the latest applications into kubernetes cluster, production DR can be live within 2 hours. Level 16 | Grand Ballroom I Jiaxin Shan, Liguang Xie Managing large-scale LLM inference workloads on Kubernetes requires more than just high-performance inference engines like vLLM. It demands a comprehensive control plane that integrates deeply with engines while addressing the complexities of large-scale operations. This need inspired the creation of AIBrix, a Kubernetes-native control plane designed to scale LLM inference with modularity, flexibility, and cutting-edge algorithms. AIBrix introduces a pluggable architecture with components for LLM specific autoscaling, high-density lora management, distributed KV cache, heterogenous serving, model loading etc. AIBrix emphasizes deep co-design with inference engines, enabling advanced features and optimizations. This talk will demonstrate AIBrix in action, showcasing its ability to improve scalability and optimize resource utilization. Additionally, we will present detailed benchmarks to evaluate the performance of these components, providing actionable insights for practitioners. Level 19 | Crystal Court I Cyrus Wong, Hongyi Qian Kubernetes Isekai (異世界） is an open-source RPG designed for hands-on Kubernetes learning through gamification. Ideal for junior to Higher Diploma students at Hong Kong Institute of Information Technology (HKIIT), it transforms Kubernetes education into an engaging adventure. Role-Playing Adventure: Students interact with NPCs who assign Kubernetes tasks.
Task-Based Learning: Tasks involve setting up and managing Kubernetes clusters.
Free Access: Uses AWS Academy Learner Lab with Minikube or Kubernetes.
Scalable Grading: AWS SAM application tests Kubernetes setups within AWS Lambda.
Progress Tracking: Students track progress and earn rewards.
This game offers practical Kubernetes experience in a fun, cost-effective way.
GenAI Chat: Integrates Generative AI to make NPC interactions more dynamic and fun, enhancing the overall learning experience.
Demo
https://www.youtube.com/watch?v=dIwNWwz681k Level 16 | Grand Ballroom I Vec Sun, Yang Song Search, advertising, and recommendation services are among the primary business types within Xiaohongshu. Due to the strong dependency of these services on index table, each instance replica needs to maintain its own independent state. As a result, such services are deployed using the stateful workload.
With the rapid growth of Xiaohongshu's business scale, the size limit of a single Kubernetes cluster has made it impossible to further scale stateful services. To address daily traffic and business growth, the only solution was to migrate workloads to idle clusters. However, this migration approach has caused significant inconvenience and risks for developer.
To tackle this challenge, Xiaohongshu leveraged Karmada to implement the federation of stateful services. By designing scheduling and deployment capabilities for stateful services on federated clusters, This approach has seamlessly resolved the scaling limitations caused by single-cluster capacity constraints for stateful services. Level 16 | Grand Ballroom I Yuji Liu, Zhiheng Sun Kuaishou's Kubernetes-based platform manages 200,000+ machines and 10M+ Pods, generating 10TB+ daily data. AI-driven intelligent portrait service enhances stability and performance:
● Stability Management: AI analyzes system and workload metrics to generate machine health scores, integrated into Kubernetes scheduling to evict/avoid unhealthy nodes. This reduced pod creation delays from 20 to 0.1 cases/day and boosted service availability from 90% to 99.99%.
● Performance Optimization:
Serving 10,000+ services with diverse resource sensitivities (compute-, cache-, and IO-intensive), we combine AI with microarchitecture data to pinpoint bottlenecks and create application profiles. Optimizing resource allocation (compute, cache, memory bandwidth) has increased average IPC by 20% and reduced LLC miss rates for cache-sensitive services from over 50% to 10%.
Future plans include integrating AI Agent technology to automate anomaly detection and reduce manual operations by 80%. Level 19 | Crystal Court I Kevin Wang, Long Xu As China's leading video platform, Bilibili faces 4 key challenges in multi-cluster AI workloads management:
1. Workload Diversity: Training/inference/video processing workloads have different scheduling requirements.
2. Cross-Cluster Complexity: Managing workloads across multiple Kubernetes clusters in expanding IDCs with SLAs.
3. Performance Demands: Minimal startup latency and best scheduling efficiency for short-running tasks e.g. video processing.
4. Efficiency-QoS Balance: maximizing resource utilization while ensuring priority workload stability. This talk will share experiences and delve specific optimization techniques:
1. Leveraging and optimizing CNCF projects such as Karmada and Volcano to build a unified, high-performance AI workload scheduling platform.
2. Integrating technologies such as KubeRay to schedule various AI online and offline workloads.
3. Maximizing resource efficiency through online and offline hybrid scheduling, tidal scheduling and other technologies. Level 16 | Grand Ballroom I Jianlin Lv, Liyi Huang Kubernetes admins often struggle to understand pod activities, both for regular pods and those with various privileges. This session explores two use cases that highlight why Tetragon, an eBPF-based observability and enforcement tool, for pod security:
1.Replacing Auditbeat with Tetragon: Learn how Auditbeat rules mapped to Tetragon tracing policies, identifying functionality gaps, and how eBay contributed back to the community
2.Auditing Container Process Permissions: See how Tetragon helped analyze pod behavior and determine if applications could migrate to more restrictive pod security policies, ensuring adherence to the principle of least privilege
We also cover deployment challenges, such as integrating with SIEM platforms, resource utilization, and implementing runtime enforcement for unwanted pod behavior. This talk provides practical insights into using Tetragon for observability, policy refinement, and improving overall pod security posture in Kubernetes environments. Level 16 | Grand Ballroom I Vivian Hu, Lv Yi While SaaS AI providers like OpenAI offer convenient LLM services, they come with significant drawbacks: high costs, lack of customization, lack of privacy, and usage limitations that can throttle high-volume applications. This presentation shows how a leading e-commerce web site deployed a highly customized suite of LLM applications on private cloud infra, reducing costs by 90% while maintaining complete control over scalability and quality of service. We'll discuss the technology stack for orchestrating inference workloads on cloud GPUs, and explore practical strategies for building stable, scalable, high-performance AI apps on your own private cloud infra. Level 16 | Grand Ballroom I Ilan Filonenko, Michas Szacillo Bloomberg’s Data Analytics Platform Engineering team supports a wide-range of real-time streaming, large batch ETL, and data exploration use-cases by using Apache Flink, Apache Spark, and Trino across multi-cluster Kubernetes. However, deploying and managing these workflows at scale efficiently can be challenging due to varying resource requirements and uptime needs. For stateful applications like Apache Flink, ensuring recovery and state conservation after downtime is especially important. This session will discuss how Bloomberg uses Karmada, a multi-cluster management system, to deploy and manage Apache Flink. We’ll also explore how Karmada’s capabilities can be expanded to handle additional data analytics workloads, including Apache Spark and Trino. The session will cover the unique requirements and real-life use-cases for each, including: - Resource-aware workload scheduling
- Custom resource requirements and health interpretation
- State conservation during application failover Level 19 | Crystal Court II Casey Li, Jing Shang As large language model (LLM) applications are widely deployed, their complex architectures challenge business observability. APM probes, which rely on instrumentation or proxy operation, consume system resources and impact traffic and performance, restricting their use in complex scenarios. Also, multiple teams handling different LLM instances make it hard to coordinate unified observability construction.
To solve this, China Mobile‘'s Panji platform collaborates with DeepFlow to achieve zero-intrusion (Zero Code) and full-stack (Full Stack) observability instantly, using eBPF and Wasm technologies. eBPF collects real-time data at the kernel level, while Wasm plugins parse streaming requests. By integrating existing data, the platform provides service universal map, distributed tracing, and multi-dimensional metric analysis, ensuring the stability and performance optimization of LLM applications. Level 16 | Grand Ballroom I Rong Zhang, Kay Yan Kubespray, recognized by Kubernetes' SIG Cluster Lifecycle, deploys production-ready Kubernetes clusters on bare metal, enhancing performance for AI applications with robust GPU support. This session covers Kubespray's fundamentals, key features, and updates. As AI workloads like LLMs grow, scalable GPU clusters are essential. Engineers will share insights from deploying custom GPU clusters at scale with Kubespray, discussing challenges and best practices. Attendees will learn to integrate Kubernetes technologies like LWS, Kueue, Gateway API Inference Extension, DRA, and tensor parallelism to enhance AI workloads like RAG and LoRA, improving resource utilization and performance. We'll share Kubespray's inventory source code to customize AI clusters and use Kubernetes operators to define infrastructure in private clouds, enabling efficient cluster scaling. Level 21 | Pearl Pavilion Yang Cao Training trillion-parameter AI models requires significant GPU resources, where any idle time leads to increased costs. Maintaining full-speed GPU utilization is crucial, yet hardware and software failures (such as firmware, kernel, or hardware issues) often disrupt large-scale training. For example, LLaMA3 experienced 419 interruptions over 54 days, with 78% due to hardware issues, underscoring the necessity for automated anomaly recovery.
At Ant Group, we will share:
GPU Monitoring: Comprehensive monitoring from hardware to applications to ensure optimal performance.
Self-Healing for Large GPU Clusters: Automated fault isolation, recovery from kernel panics, and node reprovisioning for clusters with 10,000+ GPUs.
Core Service Level Objectives (SLOs): Achieving over 98% GPU availability and more than 90% automatic fault isolation.
Predictive Maintenance: Using failure pattern analysis to reduce downtime and improve reliability. Level 19 | Crystal Court II Yanjun Chen With the development of AI technology, the demand for computing power for large model training has accelerated the deployment of AI infrastructure. Data centers often have a "resource wall" problem between AI acceleration hardware of different generations and manufacturers, which caused the incompatibility issue of software and hardware stack. Thus, it’s a big challenge for AI infra operators to maximize resource utilization. This topic focuses on technical solutions for collaborative training using chips of different architectures, sharing the practices on solving key problems such as heterogeneous training task splitting, heterogeneous training performance prediction, and heterogeneous hybrid communication and etc.. The project has been open sourced and will be further improved with better maturity through the community. Level 21 | Emerald PavilionTuesday, June 10
Keynote: Scaling Model Training with Volcano: iFlytek’s Kubernetes Breakthrough
Abstract
Keynote: The Future of AI in Hong Kong: From Local Innovation to Global Influence
Abstract
AI Model Distribution Challenges and Best Practices
Abstract
An Alternative Metadata System for Large Kubernetes Clusters
Abstract
Defining a Specification for AI/ML Artifacts
Abstract
Fast and Furious: Practice in Horizon Robotics on Large-scale End-to-end Model Training
Abstract
Unlocking Kyverno: Mastering Policy Management in Large-Scale Kubernetes Clusters
Abstract
Simplifying the Networking and Security Stack With Cilium, Hubble, and Tetragon
Abstract
⚡ Lightning Talk: Disaster Recovery - How IaCaC and Kubernetes Enables Cost Efficiency and Fast Recovery
Abstract
Introducing AIBrix: Cost-Effective and Scalable Kubernetes Control Plane for VLLM
Abstract
⚡ Lightning Talk: Kubernetes Isekai (異世界）：Transforming Kubernetes Education Into a Gamified Adventure
Abstract
⚡ Lightning Talk: Stateful Service Federation in Large-Scale Search, Ads, and Recommendation Scenarios at Xiaohongshu
Abstract
Portrait Service: AI-Driven PB-Scale Data Mining for Cost Optimization and Stability Enhancement
Abstract
Wednesday, June 11
Keynote: Optimizing AI Workload Scheduling: Bilibili's Journey To an Efficient Cloud Native AI Platform
Abstract
Keynote: Who Owns Your Pod? Observing and Blocking Unwanted Behavior at eBay With eBPF
Abstract
Keynote: How We Save $900 per Day with Self-Hosted AI: Building Scalable Local LLM Infrastructure
Abstract
How Bloomberg Creates a Resilient Data Analytics Platform Using Karmada
Abstract
China Mobile's Panji Platform: Observability Practices and Implementations for LLM Applications Base
Abstract
Building Custom GPU Clusters at Scale: Using Kubespray To Create High-Performance AI Infrastructure
Abstract
Stability in Large Model Training: Practices in Software and Hardware Fault Self-Healing
Abstract
Heterogeneous Hybrid Distributed Training for Large-Scale Language Models
Abstract
Event pages and schedules are maintained by the Linux Foundation.