The main challenge for Zillion was to deal with the huge datasets, including the consolidated medical records and participant data from the devices they already use, from smartphones to Fitbits and more. The existing pipelines were processing and loading the whole datasets (in terabytes) each time when the pipeline ran which was taking quite a long time. The challenge was to build the pipeline to implement the deltas so that the data processing and loading occur only on the changed data (inserted/updated) instead of the whole dataset. Another challenge was to reshape the existing unstructured code base using scala best practices and converting the application to microservices-based architecture.