A Future State Big Data Solution For a Big Data Giant

Clarity Insights Big Data Consultants Help Improve Performance

Challenge

A System Slowing Everyone Down 

This client had more than 1 Billion customers a day interacting with 2M advertisers on this site. This generated a lot of data; 22 terabytes to be precise. 

As a result, the client has hundreds of petabytes of raw data stored in Hadoop with multiple terabytes of new data ingested every hour. Fast-paced growth and increasing data ingestion were causing data management challenges. In order to scale and meet growing business needs, the client needed a better solution.

 

Solution

Rearchitecting the system

In order to improve the system's performance, Clarity:

  • Architected scalable, end-to-end processes to consume large volumes of complex data from Hive, Scribe and3P APIs
  • Integrated datasets into Hive, Vertica and 3P APIs
  • Built data integration pipelines for advertising campaigns
  • Debugged Big Data ETL pipelines
  • Developed custom UDFs for data transformation

Outcome

Faster, cheaper, better 

  • Significant cost savings compared with traditional, legacy environment.
  • Development of a next-generation platform for user insight and ad-hoc analytics.
  • Support for EDW-class, structured analytics for multiple petabytes on a multi-node EDW cluster.
  • Sample analytics, including a monetization roadmap to increase demand in the social media platform and insight to increase revenue from existing customers through media-mix optimization.
Contact us

Technologies

vertica_logo_1.png
hadoop.jpeg

Latest News

Why Patients are Fleeing Health Systems and How Providers Can Fix It with Patient Analytics

From disease awareness to post-treatment discharge, patients pay close attention to how their interactions with the healthcare system are making their care journey either seamless or disjointed. As

Corporate Lifesaver: Leveraging Data Governance Strategy for Market Differentiation

Data governance strategy is something every organization has started to talk about - though levels of approach or maturity down that path vary. Customers will often ask us “How do I sell a long-term

Remembering the Easily Forgettable: Why Persistent Memory Servers will Change BI

For machines loaded with so much memory, computers can be terribly forgetful. All they need is a simple shutdown to lose every piece of information you loaded into them. Then, it’s,  “Bye-bye. No

why-clarity.jpg

Find out how Clarity Insights can help you 

Contact us