A Future State Big Data Solution For a Big Data Giant

This client had more than 1 Billion customers a day interacting with 2M advertisers on this site. This generated a lot of data; 22 terabytes to be precise. 

As a result, the client has hundreds of petabytes of raw data stored in Hadoop with multiple terabytes of new data ingested every hour. Fast-paced growth and increasing data ingestion were causing data management challenges. In order to scale and meet growing business needs, the client needed a better solution.

In order to improve the system's performance, Clarity:

• Architected scalable, end-to-end processes to consume large volumes of complex data from Hive, Scribe and3P APIs
• Integrated datasets into Hive, Vertica and 3P APIs
• Built data integration pipelines for advertising campaigns
• Debugged Big Data ETL pipelines
• Developed custom UDFs for data transformation

• Significant cost savings compared with traditional, legacy environment.
• Development of a next-generation platform for user insight and ad-hoc analytics.
• Support for EDW-class, structured analytics for multiple petabytes on a multi-node EDW cluster.
• Sample analytics, including a monetization roadmap to increase demand in the social media platform and insight to increase revenue from existing customers through media-mix optimization.