Hybrid Cloud-Native AI Architecture for Scalable Deployment of Agentic and Generative AI Applications
Abstract
Agentic and generative AI applications introduce workload characteristics -- long-lived multi-step sessions, heterogeneous tool invocation, bursty and unpredictable demand, and mixed regulatory requirements over data and models -- that strain cloud-native infrastructure patterns originally designed for stateless microservices. This paper presents a hybrid cloud-native architecture that spans public cloud and on-premises infrastructure to meet the latency, cost, reliability, and data-governance requirements of production agentic AI systems. We describe a layered reference architecture comprising an agent orchestration mesh, a cascade-routed model serving layer, a federated data and retrieval layer, and a Kubernetes-based multi-cluster control plane, and we evaluate it empirically against single-environment deployment baselines. Across a range of experiments spanning latency, cost, GPU utilization, autoscaling responsiveness, and fault injection, the hybrid architecture with policy-based routing reduces median request latency by 34%, lowers cost per 1,000 agent tasks by 57% relative to a cloud-only baseline, and sustains 93% task success under simulated cross-region network partition versus 56% for a single-region deployment. We conclude with a discussion of open challenges in multi-agent state management, cross-environment observability, and the governance of autonomous tool-calling agents operating across trust boundaries.
References
[1] Burns, B., Grant, B., Oppenheimer, D., Brewer, E., & Wilkes, J. (2016). Borg, Omega, and Kubernetes. Communications of the ACM.
[2] Verma, A., Pedrosa, L., Korupolu, M., Oppenheimer, D., Tune, E., & Wilkes, J. (2015). Large-scale cluster management at Google with Borg. Proceedings of the European Conference on Computer Systems.
[3] Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., et al. (2023). Efficient memory management for large language model serving with PagedAttention. Proceedings of the ACM Symposium on Operating Systems Principles.
[4] Zhong, Y., Liu, S., Chen, J., Hu, J., Zhu, Y., Liu, X., et al. (2024). DistServe: Disaggregating prefill and decoding for goodput-optimized large language model serving. Proceedings of the USENIX Symposium on Operating Systems Design and Implementation.
[5] Chen, L., Zaharia, M., & Zou, J. (2023). FrugalGPT: How to use large language models while reducing cost and improving performance. arXiv preprint.
[6] Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., & Cao, Y. (2023). ReAct: Synergizing reasoning and acting in language models. Proceedings of the International Conference on Learning Representations.
[7] Wu, Q., Bansal, G., Zhang, J., Wu, Y., Li, B., Zhu, E., et al. (2023). AutoGen: Enabling next-gen LLM applications via multi-agent conversation. arXiv preprint.
[8] Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., et al. (2020). Retrieval-augmented generation for knowledge-intensive NLP tasks. Advances in Neural Information Processing Systems.
[9] Newman, S. (2021). Building Microservices: Designing Fine-Grained Systems (2nd ed.). O'Reilly Media.
[10] Morris, K. (2020). Infrastructure as Code: Dynamic Systems for the Cloud Age (2nd ed.). O'Reilly Media.
[11] Basiri, A., Behnam, N., de Rooij, R., Hochstein, L., Kosewski, L., Reynolds, J., & Rosenthal, C. (2016). Chaos engineering. IEEE Software.
[12] Sculley, D., Holt, G., Golovin, D., Davydov, E., Phillips, T., Ebner, D., et al. (2015). Hidden technical debt in machine learning systems. Advances in Neural Information Processing Systems.
[13] Patel, P., Choukse, E., Zhang, C., Shah, A., Goiri, I., Maleki, S., & Bianchini, R. (2024). Splitwise: Efficient generative LLM inference using phase splitting. Proceedings of the International Symposium on Computer Architecture.
[14] Ghemawat, S., Gobioff, H., & Leung, S. T. (2003). The Google File System. Proceedings of the ACM Symposium on Operating Systems Principles.
[15] Villalobos, P., Ho, A., Sevilla, J., Besiroglu, T., Heim, L., & Hobbhahn, M. (2024). Position: Will we run out of data? Limits of LLM scaling based on human-generated data. Proceedings of the International Conference on Machine Learning.
[16]Konda, P. R. (2023). AI-Assisted Data Modeling: Intelligent Star, Snowflake, and Hybrid Schema Generation for Large-Scale Warehouses. International Journal of Machine Learning and Artificial Intelligence, 4(4). https://jmlai.in/index.php/ijmlai/article/view/90
[17]Ensuring BI Reporting Accuracy Using AI-Based Back-Tracing of Metrics to ETL Lineage and Data Marts. (2023). International Machine Learning Journal and Computer Engineering, 6(6). https://mljce.in/index.php/Imljce/article/view/69
[18]AI-Driven Intelligent Data Anomaly Detection Using Machine Learning Techniques. (2024). International Machine Learning Journal and Computer Engineering, 7(7). https://mljce.in/index.php/Imljce/article/view/71
[19]Konda, P. R. (2023). Mining Data Lineage Patterns Using Machine Learning to Predict Downstream Impact. International Meridian Journal, 5(5). https://meridianjournal.in/index.php/IMJ/article/view/117
[20]Konda, P. R. (2024). Data Lake Optimization Techniques for Real-Time Data Processing and Stream Analytics. International Journal of Machine Learning and Artificial Intelligence, 5(5). https://jmlai.in/index.php/ijmlai/article/view/91
