preview

Dim Look Up Persuasive Speech

Decent Essays

Hey folks, We are excited to announce the release of the new dim lookup, which offers significant performance gains over the existing dim lookup. We will replace the existing dim lookup with the new one over the next few months. If you are using dim lookup, we request your cooperation in making the transition smooth for you and for us. This email provides a quick overview of dim lookup and the new version, followed by what you need to do to make use of the new dim lookup, and to make sure your queries and pipelines are not adversely affected by the change. What is dim lookup? Our tables can be widely classified into fact tables (measurements, e.g., impressions, clicks, conversions) and dimension tables (configuration, e.g., campaigns, lineitems, tactics). While processing data from fact tables, the requirement for related data from …show more content…

With time, our dimension data has seen explosive growth. The number of stored distinct campaign ids has grown from 611 in early 2010 to over 28,000 today and tactic ids from 1563 in early 2010 to over 260,000 today. This graph shows the growth in number of distinct advertisement ids: With such a baffling increase in the size of dimension data, our existing dim lookup, which has been in use since 2009, is not up to the job anymore. Its performance is degrading and queries using it are getting slower and require more than reasonable amounts of memory. What is the new dim lookup? Over the past year, we developed Luke, a distributed application that powers the new dim lookup UDF and exposes a Java interface for programmatic access of dimension data. We have already switched to the new dim lookup in our Apollo/Fuel reporting and as a result, the memory requirement has gone down 4 times (from 8 GB to 2 GB) and the running time has gone down by more than 33% (from 12+ hours to approx. 8 hours). For details about the internal structure of Luke, refer to this document. When is it coming to

Get Access