Spotlight on Tech

Solving the challenges of deployment and management of complex analytic pipeline on Kubernetes

By
Mehran Hadipour
VP - BD & Tech Alliances
Rakuten Symphony
November 13, 2023
5
minute read

Kubernetes (K8s) can be an excellent application deployment platform. However, analytics pipelines can provide unique challenges including complexity in deployment data management and lifecycle operations. Dell and Rakuten Symphony, enabled by Intel technology, have collaborated to radically simplify the delivery of complex data analytics applications for both the data scientist and the IT professional, which is detailed in a new solution brief from Intel. 

The solution combines selected Dell storage and server platforms coupled with Application-as-a-Service capability of Rakuten Symphony Symcloud that eliminate complexity, accelerate deployment and automate lifecycle operations of applications, including horizontal and vertical scaling, upgrade automation and performance management. 

The solution incorporates software-defined storage that provides data protection services including snapshot and clones that are application aware and thin provisioned. This includes application consistency across all volumes used by the application pipeline as a whole, as well as K8 configuration including secrets and labels that enables recovery to any point in time of the entire stack. 

Provisioning of applications and workloads 

Provisioning a diverse set of applications to deliver the entire pipeline is complicated by managing multiple helm charts and considerations and infrastructure dependencies including networking, storage allocation and complex affinity and anti-affinity policies that must be programed. 

Performance management and scaling to meet demands

After the initial deployment, it is quite complex to manage competing requirements and resources allocated to the application to ensure proper performance. Scaling and healing operations are complicated by the fact that there are multiple stateful application components that must be managed to achieve the desired consistent performance. 

Managing data storage lifecycle 

Analytics applications are stateful and running them on Kubernetes, which was designed for stateless workload, requires storage services and container storage interface (CSI) to be used to perform basic operations including replication, snapshots, clone and backup, and business continuity and disaster recovery (BCDR). Since the applications run across multiple POD with many different volumes attached, maintaining application-level consistency for data protection is necessary, however achieving this is complex as the storage layer needs to be application aware. 

Complexity of day 2 operations for the applications and infrastructure 

It is particularly complex and time consuming to conduct the initial deployment of the analytics application and infrastructure management for day 2 operations. This includes infrastructure upgrade and scaling, applications upgrades, healing, and scaling and life cycle management of the end-to-end application delivery infrastructure especially when it is being used across multiple tenets. 

Analytics as a service solution with Dell Data Lakehouse 

To solve these challenges, an agile analytics-as-a-service platform is developed by collaboration between Dell and Rakuten Symphony. The solution is a pre-qualified stack of Intel-based hardware coupled with Symcloud Kubernetes platform that offers self-service, rapid provisioning, simplified management, and performance optimization. The solution is designed to meet the requirements of both data scientists and IT/Cloud professionals.

Business needs and requirements from an AI platform for data scientists and IT/Ops

Application Provisioning

Rakuten Symphony Symcloud platform provides an app store capability. The application profiles can be used by the data scientist to self-provision the pipeline and the container management system provisions the required storage and compute infrastructure automatically from the available cluster resources.  

Symcloud app store capability — Application-as-a-Service.
Symcloud app store capability — Application-as-a-Service.

Day 2 life cycle operations  

Post deployment of the application pipeline many of the complex day 2 operations are automated, including data management aspects of the application pipeline.  

The application profiles have been set up to specify automated processes around data protection including data at rest encryption, Replication (protect from storage failure), Snapshot (protect against corruption and end user errors) and backup and recovery. 

Lifecycle operations of applications can also be automated including horizontal and vertical scaling, upgrade automation and performance management at the application level. 

The data protection services including snapshot and clones are application aware and thin provisioned. This includes cross application consistency across all volumes used by the application pipeline as a whole, as well as K8 configuration including secrets and labels that enables recovery to any point in time of the entire stack. 

This greatly simplifies data management post deployment and removes the requirements of additional technical storage resources. The software-defined storage stack will absorb any type of storage attached to the cluster seamlessly and provision it across the required applications optimally, considering IOOPS and QOS requirements stated in the application profile. 

Dell Data Lakehouse proven solution with Dell and Rakuten Symphony. 
Dell Data Lakehouse proven solution with Dell and Rakuten Symphony. 

Pretested and proven solution 

There are complex infrastructure dependencies when building a private cloud for analytics including storage and networking performance as well as compute resources required. 

Dell Data Lakehouse is a pretested platform with a choice of compute and storage configurations that is aligned with Intel technology and tested to deliver scale and performance for the selected set of applications. 

For more information about Dell solution with Rakuten Symphony, please download the Private Cloud Analytics solution brief and existing white paper with Dell.  And for more on the Symcloud platform and to get in touch with us, please visit our product page

Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.
Kubernetes
Kubernets Storage
Symcloud
How can Symphony help?
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.
By clicking “Accept All Cookies”, you agree to the storing of cookies on your device to enhance site navigation, analyze site usage, and assist in our marketing efforts. View our Privacy Notice for more information.