A Future State Big Data Solution For a Big Data Giant

Clarity Insights Big Data Consultants Help Improve Performance

Challenge

A System Slowing Everyone Down 

This client had more than 1 Billion customers a day interacting with 2M advertisers on this site. This generated a lot of data; 22 terabytes to be precise. 

As a result, the client has hundreds of petabytes of raw data stored in Hadoop with multiple terabytes of new data ingested every hour. Fast-paced growth and increasing data ingestion were causing data management challenges. In order to scale and meet growing business needs, the client needed a better solution.

 

Solution

Rearchitecting the system

In order to improve the system's performance, Clarity:

  • Architected scalable, end-to-end processes to consume large volumes of complex data from Hive, Scribe and3P APIs
  • Integrated datasets into Hive, Vertica and 3P APIs
  • Built data integration pipelines for advertising campaigns
  • Debugged Big Data ETL pipelines
  • Developed custom UDFs for data transformation

Outcome

Faster, cheaper, better 

  • Significant cost savings compared with traditional, legacy environment.
  • Development of a next-generation platform for user insight and ad-hoc analytics.
  • Support for EDW-class, structured analytics for multiple petabytes on a multi-node EDW cluster.
  • Sample analytics, including a monetization roadmap to increase demand in the social media platform and insight to increase revenue from existing customers through media-mix optimization.
Contact us

Technologies

vertica_logo_1.png
hadoop.jpeg

Latest News

What Will Augmented Data Management Mean for Your Organization?

To understand augmented data management, it’s first necessary to know what is involved in basic data management. “Data management” is an overly broad term relating to activities ranging from database

The Customer is Always...Oh, Never Mind...The Art of a Customer Data Strategy

The customer is always right. Uh, no... what if the customer wants something that’s bad for your business? Then, that customer is right… and wrong. Problem is, how will you know? This is what having

No More Data Fabrics. It’s Time for the Data Trampoline.

Another day, another data management metaphor... Gartner, for example, cited the “data fabric” as one of their “Top 10 Data and Analytics Technology Trends That Will Change Your Business.” The term

why-clarity.jpg

Find out how Clarity Insights can help you 

Contact us