A Future State Big Data Solution For a Big Data Giant

Clarity Insights Big Data Consultants Help Improve Performance

Challenge

A System Slowing Everyone Down 

This client had more than 1 Billion customers a day interacting with 2M advertisers on this site. This generated a lot of data; 22 terabytes to be precise. 

As a result, the client has hundreds of petabytes of raw data stored in Hadoop with multiple terabytes of new data ingested every hour. Fast-paced growth and increasing data ingestion were causing data management challenges. In order to scale and meet growing business needs, the client needed a better solution.

 

Solution

Rearchitecting the system

In order to improve the system's performance, Clarity:

  • Architected scalable, end-to-end processes to consume large volumes of complex data from Hive, Scribe and3P APIs
  • Integrated datasets into Hive, Vertica and 3P APIs
  • Built data integration pipelines for advertising campaigns
  • Debugged Big Data ETL pipelines
  • Developed custom UDFs for data transformation

Outcome

Faster, cheaper, better 

  • Significant cost savings compared with traditional, legacy environment.
  • Development of a next-generation platform for user insight and ad-hoc analytics.
  • Support for EDW-class, structured analytics for multiple petabytes on a multi-node EDW cluster.
  • Sample analytics, including a monetization roadmap to increase demand in the social media platform and insight to increase revenue from existing customers through media-mix optimization.
Contact us
technologies

Technologies

vertica_logo_1.png
hadoop.jpeg

Latest News

For Healthcare Organizations, a Modern Data Architecture Is a Necessity

Healthcare Partner Bob Fuller and Healthcare Chief Technology Officer Ramu Kalvakuntla make a compelling argument for modern data architectures for healthcare organizations. Not sure what those are?

Key Steps to Success with Self-Service Analytics

Caitlin Willich knows a thing or two about self-service analytics, having helped Fortune 500 clients across the nation implement successful projects. In this piece featured in Information Management,

Disruption is the Wake-Up Call Healthcare Needs

Health Innovation Lead Max Schlather sees Silicon Valley giving the healthcare industry a needed wake-up call. He shared his thoughts in an editorial on Healthcare Analytics News. Check it out below.

why-clarity.jpg

Find out how Clarity Insights can help you 

Contact us