Technical insight into Big Data, Cloud, Graph & Artificial Intelligence technologies

When designing data pipelines, it is a common requirement to know if the data required for processing within a particular timeframe is available or not. Mostly this timeframe is one day for daily incremental processes, where a daily ETL process wants to know, if the data for current date is even available for processing or not.  For instance a process executing daily, if the data for current date is loaded into the source directory where it reads from.  If you are using Hadoop's HDFS for your data storage, in this article I'll share a piece of code, to easily verify if the data for a desired date is present in a hive table(which always have an underlying directory in HDFS).  Let us take a real world scenario where we get the data loaded in automated fashion, into a certain hive table, which would always have an underlying directory in HDFS. This data load can be through a sourcing ingestion process or the load as a result of an ETL process or the outcome of an...

Since Hadoop's inception, we have learnt that it is good for processing small number of large files, instead of large number of small files. Part of the reason pertaining to this behavior is, its MapReduce Paradigm which works on splits of input data.  In our day to day data processing, there can be various situations where our processes can produce large number of small files. Assume a process generated output data of 300 MB, but this data is distributed in approximately 3000 files of 99 KB each. Today we'll learn how to merge such large number of small files into bigger files. Before that let us quickly understand the problems, such large number of small files can cause. Problems posed by large number of small files -   1.   I f we store the data of this table in GZIP format(3000 files each of 99KB), then a downstream process running a hive query(MapReduce job) on this data, would spin up 3000 mappers to process individual splits for each file, as the GZIP format ...

It is a well known fact, that parquet format is a binary format. And for humans, reading binary data is not as easy as opening a text file and understanding its contents. But no worries, today we'll learn how to read a parquet file . If you would like to understand all the internal details of a parquet file , you can please refer my previous article here . In this article I have tried to explain all the inner storage details of Parquet and answered most of the questions around its usage, efficiency and performance.                                                                                Parquet tools - As we know parquet is a binary format, we need an interpreter to translate its contents so that we can understand them. And this is what parquet-tools  utility project does...

Can I do row level updates in Hive? We are all aware of UPDATE not being a supported operation, based on past years of our experience of Hive. But guess what, it is no more a constraint, as UPDATE is available starting Hive 0.14 version.  In this post we'll learn about the details of UPDATE in Hive(a long awaited operation for hadoop platform, as required by most of the Big data Engineers). We will learn about the following details: 1. The prerequisites for hive to perform update. 2. Syntax of update. 3. Hive configuration settings to do update. 4. A close look at what happens at Hadoop file system level when update operation is performed. 5. Limitations to UPDATE operation in Hive For a hive table to be updated(as of the date of publishing this article) : 1. It should be stored as ORC file format . 2. It should be bucketed . 3. It should support transactions(ACID semantics enabled) . Standard Syntax: UPDATE tablename SET column = value [, column = v...

The data is growing swiftly and with its growth it brings more opportunities of analysis, to derive more value and business. The growth rate is astonishing and in itself justifies why organizations across the globe are moving towards Big Data Solution like Hadoop. Following are some of the Forbes Facts(as of the date of publishing this article), which defines the pace at which the data is increasing and provides us a definite reason, for why the world is moving towards a Big Data solution for managing data: More data has been created in the past 2 years than in the entire previous history of human race. Data is growing faster than ever before and by the year 2020, about 1.7 megabytes of new information will be created every second for every human being on the planet.  By 2020, our accumulated digital universe of data will grow from 4.4 zettabyets today to around 44 zettabytes, or 44 trillion gigabytes. Every second we create new data. We perform approx 40,000 search que...

Columnar storage has revolutionized Big data processing, since its inception.  Its power can be realized from the fact that Google Big Query, Hbase, Amazon Redshift, Azure SQL Data Warehouse and many more, all utilize columnar storage. As you are here today reading this article, it is obvious that you are curious to learn how Columnar storage internally works. Well, just stay tuned for next few minutes, as  I'll explain all the details of the  most popular columnar file format called Parquet.  We'll learn the details  by  actually opening  a parquet file and by  going deep, at the level of how things work under the hood at disc level.  We'll also learn why Parquet would save cost, when it is used as underlying file format with A WS services such as Athena . When to use Parquet format  -  Just check this less than 30 seconds conversation, going on between a Data Engineer(male) and a  BI Analyst(female) . When to us...