Skew Join Optimization in Hive
Skew is a very common issue which most of the data engineers come across. What is Skew - When in our data we have very large number of records associated with one(or more) particular key, then this data is said to be skewed on that key. We'll understand this with a very simple example. Think about world's population data table, where each record is of one person and the number of records equals to total world population. As we all know China and India, each accounts for around 17-18% of world's population, hence such population table is referred to as skewed on keys country=China and country=India. If you focus on the following pie chart of world's population distribution by country, then this would make exact sense. Problem statement - Assume we have a table which tracks the online visits to different online websites. In this table amazon.com being very popular, has 1 Billion rows whereas other not very popular sites ( www.abc.com , w...