Kubernetes (K8s) can be an excellent application deployment platform. However, analytics pipelines can provide unique challenges including complexity in deployment data management and lifecycle operations. Dell and Rakuten Symphony, enabled by Intel technology, have collaborated to radically simplify the delivery of complex data analytics applications for both the data scientist and the IT professional, which is detailed in a new solution brief from Intel.
The solution combines selected Dell storage and server platforms coupled with Application-as-a-Service capability of Rakuten Symphony Symcloud that eliminate complexity, accelerate deployment and automate lifecycle operations of applications, including horizontal and vertical scaling, upgrade automation and performance management.
The solution incorporates software-defined storage that provides data protection services including snapshot and clones that are application aware and thin provisioned. This includes application consistency across all volumes used by the application pipeline as a whole, as well as K8 configuration including secrets and labels that enables recovery to any point in time of the entire stack.
Provisioning a diverse set of applications to deliver the entire pipeline is complicated by managing multiple helm charts and considerations and infrastructure dependencies including networking, storage allocation and complex affinity and anti-affinity policies that must be programed.
After the initial deployment, it is quite complex to manage competing requirements and resources allocated to the application to ensure proper performance. Scaling and healing operations are complicated by the fact that there are multiple stateful application components that must be managed to achieve the desired consistent performance.
Analytics applications are stateful and running them on Kubernetes, which was designed for stateless workload, requires storage services and container storage interface (CSI) to be used to perform basic operations including replication, snapshots, clone and backup, and business continuity and disaster recovery (BCDR). Since the applications run across multiple POD with many different volumes attached, maintaining application-level consistency for data protection is necessary, however achieving this is complex as the storage layer needs to be application aware.
It is particularly complex and time consuming to conduct the initial deployment of the analytics application and infrastructure management for day 2 operations. This includes infrastructure upgrade and scaling, applications upgrades, healing, and scaling and life cycle management of the end-to-end application delivery infrastructure especially when it is being used across multiple tenets.
To solve these challenges, an agile analytics-as-a-service platform is developed by collaboration between Dell and Rakuten Symphony. The solution is a pre-qualified stack of Intel-based hardware coupled with Symcloud Kubernetes platform that offers self-service, rapid provisioning, simplified management, and performance optimization. The solution is designed to meet the requirements of both data scientists and IT/Cloud professionals.
Rakuten Symphony Symcloud platform provides an app store capability. The application profiles can be used by the data scientist to self-provision the pipeline and the container management system provisions the required storage and compute infrastructure automatically from the available cluster resources.
Post deployment of the application pipeline many of the complex day 2 operations are automated, including data management aspects of the application pipeline.
The application profiles have been set up to specify automated processes around data protection including data at rest encryption, Replication (protect from storage failure), Snapshot (protect against corruption and end user errors) and backup and recovery.
Lifecycle operations of applications can also be automated including horizontal and vertical scaling, upgrade automation and performance management at the application level.
The data protection services including snapshot and clones are application aware and thin provisioned. This includes cross application consistency across all volumes used by the application pipeline as a whole, as well as K8 configuration including secrets and labels that enables recovery to any point in time of the entire stack.
This greatly simplifies data management post deployment and removes the requirements of additional technical storage resources. The software-defined storage stack will absorb any type of storage attached to the cluster seamlessly and provision it across the required applications optimally, considering IOOPS and QOS requirements stated in the application profile.
There are complex infrastructure dependencies when building a private cloud for analytics including storage and networking performance as well as compute resources required.
Dell Data Lakehouse is a pretested platform with a choice of compute and storage configurations that is aligned with Intel technology and tested to deliver scale and performance for the selected set of applications.
For more information about Dell solution with Rakuten Symphony, please download the Private Cloud Analytics solution brief and existing white paper with Dell. And for more on the Symcloud platform and to get in touch with us, please visit our product page.
