Knoldus Inc

Let's Talk

nDimensional

Accelerate digital transformation journey with a unified data processing platform

Headquarters

Austin, Texas, USA

Industry

Reactive

Technology Used

Apigee Platform

nDimensional, a leading data analytics and AI company that helps companies leverage their data to get to a deeper understanding of what’s going on in their business. They needed a scalable data lake that could handle the size and complexity of a massive, growing data set. They turned to Knoldus to guide them in designing and developing a technology architecture that would be cost effective, efficient, and yet flexible enough to meet their current and future needs.

Challenges

Storing data in a unified format so that for the user there is no difference between batch and stream data.

Solutions

An interface that provides various actions for processing Batch and streaming data. Also, they help in connecting information from Kafka & other static resources like Files, DB(s), etc.

Results

Handled different sources of data like files, tables, streams (Kafka, Kinesis, etc) in the same platform.

Challenges

nD wanted to design and develop a scalable data lake that supports both Batch & Stream data that works along with their platform. Also, the objective was to make a self-containing application that can be packaged into an image (e.g., Docker) and deployed on any cloud service or on-premise.

Although, the basic flow for the application was very simple but the key challenges were:

  • Storing data in a unified format so that for the user there is no difference between batch and stream data.
  • Accessibility of data should be quick, i.e., based on user query.
  • Processing Batch/Stream using a plugable logic supplied by the user.

Solution

Knoldus worked on two key parts –

  • Fast API - An interface that provides various actions for processing Batch and streaming data. Also, they help in connecting information from Kafka & other static resources like Files, DB(s), etc.
  • Microservice Service (based on Lagom) for Background Tasks - It was developed to take load off the processing engine for saving data to the data lake (S3 and Cassandra) and assign it to a separate node. Since the service was built using Lagom, it was easily re-sizeable and deployable on different anywhere.

Results

The technology stack and architecture met the SLAs which were required for the platform. The application was best suited for reactiveness, and the entire Lightbend reactive technology stack helped achieve it. Currently, this platform is in Beta version and works like a charm.

The new platform and technologies recommended and architected by Knoldus helped nD successfully do the following:

  • Created streaming and batch pipelines along with workflows, in the same platform.
  • Handled different sources of data like files, tables, streams (Kafka, Kinesis, etc) in the same platform.
  • Supported 100,000 streams with zero latency.
  • Dramatically increase processing speed.
  • Substantially increase storage capacity.
  • Save money while scaling their business.

Explore latest Case Studies

Ready to gain a competitive advantage with Future Ready Emerging Technologies?

Let's Initiate a Partnership