AI's best kept secret

By

Photo by Kristina Flour on Unsplash

 

When you think of Artificial Intelligence, what comes to your mind ?

Most probably you think of Machine learning models or some highly sophisticated algorithms. However, behind every successful AI system is something which is far less glamorous, but definitely most critical. And that is Data.

Without strong data foundation, even the most sophisticated AI model cannot deliver accurate and reliable results. And that is why I call data engineering as the backbone and hidden secret of AI.

In this article, we will understand:

  1. Why data engineering is a true backbone of AI.
  2. How it enables AI systems to function.
  3. Last but not the least, What makes it indispensable in today’s data driven world.

1. Data is Fuel for AI model

Press enter or click to view image in full size
Data engineering, Oil refinery analogy

Every AI model needs data for its training.

But raw data as it exists in the real world, is messy, incomplete and inconsistent.

Think of it like crude oil. Crude oil has potential value, but without refinement it is waste and unusable. So we need a refinery to refine the crude oil. For data(crude oil here) Data engineering is such a refinery.

So before an AI model even receives the data for its training, it is the Data Engineering which works on:

  • First collecting data from different sources like IoT devices, logs, databases, APIs etc.
  • Cleansing data by removing duplicates, filling missing values and fixing any kind of data inconsistencies.
  • Transforming it into structured, meaningful formats that an AI model can easily understand.
  • Finally delivering this clean and transformed data at large scale in real time or batch, to feed the AI systems.

Without this preliminary process of collection, cleansing, transformation and delivery, AI models would simply be trying to just make sense of chaos.

2. Scale: Turning Massive Volume of Data into Usable Data

Press enter or click to view image in full size

Modern AI, like deep learning and large scale natural language processing, requires massive volumes of data.

But think of this:

You have Terabytes or Petabytes of data. But you cannot quickly process it. Would such Petabytes of data be of any use?

No.

It is like saying — I have an accurate AI Model, but just that it takes hours to return results.

This is exactly where Big data engineering tools, such as Apache Spark, Kafka and Cloud native tools, come to rescue. These tools enable organizations to:

  • Ingest and process vast datasets quickly.
  • Build scalable pipelines that will grow as the data volume grows.
  • Deliver low latency streaming data for real time AI applications, such as real time fraud detection or recommendation systems.

In short, data engineering transforms big data into usable data.

3. Data Quality over Data Quantity:

One of the golden rules of AI is:

If the training data is garbage (inaccurate or biased), AI model will produce equally flawed results. But Data engineers protects AI systems against this.


Before any data reaches a data science model, data engineers define and also enforce data quality checks on that data.

Press enter or click to view image in full size
AI labelling Africa as Asia and vice versa

Data engineer also builds monitoring systems that track anomalies in incoming data. They collaborate with data scientists, to ensure the datasets used for training represent high quality real world scenarios.

High quality data ensures that AI models are not just powerful, but also reliable and trustworthy.

4. Bridging the Gap Between Incoming Data and AI Models

AI models cannot exist alone. As the new data flows in, the models need to be trained, deployed and continuously retrained.

Press enter or click to view image in full size
AI model consistently training on new data

Data engineers ensures that this lifecycle keeps running smoothly by:

  • Automating pipelines that delivers fresh, relevant data to training systems.
  • Creating feature stores where reusable AI features are managed.
  • Enabling MLOps practices. These combines data engineering, machine learning and DevOps for seamless deployment.

Without this backbone, AI projects often get stuck in POC(proof of concept) mode and fail to scale into production.

5. Enabling Real-World AI Applications

Think about some of the following AI applications we use every day:

  • Netflix recommendations
  • Google Maps traffic predictions
  • Fraud detection in banking
  • Voice assistants like Alexa or Siri

All of these depend not just on models, but on robust data pipelines that performs ETL i.e. ingest(Extract), process(Transform) and deliver(Load) relevant information in real time. That is pure data engineering at work.

6. The Future: Data Engineering for AI 2.0

Press enter or click to view image in full size

As AI evolves, moving towards multimodal AI, generative models and autonomous systems — the demand for advanced data engineering will only increase. Future challenges include:

  • Ensuring ethical AI by building transparency and bias checking into data pipelines.
  • Automating more of the data lifecycle with AI driven data engineering tools.
  • Handling massive amount of unstructured data (text, video, audio) at scale.

In other words, tomorrow’s AI advancements will still stand firmly on the shoulders of strong data engineering.

Conclusion

AI may be the brain of modern technology, but data engineering is the system that keeps it alive and functioning.

Hence, the next time we hear about a revolutionary AI model, we need to remember: behind the scenes, a team of data engineers made it possible.

Thank you for reading and I hope this article offered valuable insights.

Lead Data Engineering Professional with 17+ years of experience building scalable Big Data, Cloud, Graph and AI solutions.

Comments

Post a Comment

Popular posts from this blog

UPSERT in Hive(3 Step Process)

By
Image
In this post we'll learn an efficient 3 step process, for performing UPSERT in hive on a large size table containing entire history. Just for the audience not aware of UPSERT - It is a combination of UPDATE and INSERT. If on a table containing history data, we receive new data which needs to be inserted as well as some data which is an UPDATE to the existing data, then we have to perform an UPSERT operation to achieve this. Prerequisite  – The table containing history being very large in size should be partitioned, which is also a best practice for efficient data storage, when storing large volume of data in any Big Data warehouse. Business scenario  – Lets take an example of click stream data of a website, as gathered from different browsers of visitors who visited the website. The site_view_hist table contains the clicks and page impressions counts from different browsers and the table is partitioned on hit_date(the date on which the visitor hitted or visited the websi...
Read more

Parquet tools utility

By
Image
It is a well known fact, that parquet format is a binary format. And for humans, reading binary data is not as easy as opening a text file and understanding its contents. But no worries, today we'll learn how to read a parquet file . If you would like to understand all the internal details of a parquet file , you can please refer my previous article here . In this article I have tried to explain all the inner storage details of Parquet and answered most of the questions around its usage, efficiency and performance.                                                                                Parquet tools - As we know parquet is a binary format, we need an interpreter to translate its contents so that we can understand them. And this is what parquet-tools  utility project does...
Read more

Hive - Merging small files into bigger files.

By
Image
Since Hadoop's inception, we have learnt that it is good for processing small number of large files, instead of large number of small files. Part of the reason pertaining to this behavior is, its MapReduce Paradigm which works on splits of input data.  In our day to day data processing, there can be various situations where our processes can produce large number of small files. Assume a process generated output data of 300 MB, but this data is distributed in approximately 3000 files of 99 KB each. Today we'll learn how to merge such large number of small files into bigger files. Before that let us quickly understand the problems, such large number of small files can cause. Problems posed by large number of small files -   1.   I f we store the data of this table in GZIP format(3000 files each of 99KB), then a downstream process running a hive query(MapReduce job) on this data, would spin up 3000 mappers to process individual splits for each file, as the GZIP format ...
Read more

Parquet File format - Storage details

By
Image
Columnar storage has revolutionized Big data processing, since its inception.  Its power can be realized from the fact that Google Big Query, Hbase, Amazon Redshift, Azure SQL Data Warehouse and many more, all utilize columnar storage. As you are here today reading this article, it is obvious that you are curious to learn how Columnar storage internally works. Well, just stay tuned for next few minutes, as  I'll explain all the details of the  most popular columnar file format called Parquet.  We'll learn the details  by  actually opening  a parquet file and by  going deep, at the level of how things work under the hood at disc level.  We'll also learn why Parquet would save cost, when it is used as underlying file format with A WS services such as Athena . When to use Parquet format  -  Just check this less than 30 seconds conversation, going on between a Data Engineer(male) and a  BI Analyst(female) . When to us...
Read more

Skew Join Optimization in Hive

By
Image
Skew is a very common issue which most of the data engineers come across.  What is Skew - When in our data we have very large number of records associated with one(or more) particular key, then this data is said to be skewed on that key.  We'll understand this with a very simple example. Think about world's population data table, where each record is of one person and the number of records equals to total world population. As we all know China and India, each accounts for around 17-18% of world's population, hence such population table is referred to as skewed on keys country=China and country=India. If you focus on the following pie chart of world's population distribution by country, then this would make exact sense. Problem statement  - Assume we have a table which tracks the online visits to different online websites. In this table amazon.com  being very popular,  has 1 Billion rows whereas other not very popular sites ( www.abc.com , w...
Read more

Apache Spark - Sort Merge Join

By
Image
Apache Spark is a well known open source cluster computation system, for large scale data processing. If you are into Big Data and Machine Learning, Apache Spark may not need any introduction. Its capabilities include In-memory processing, Distributed SQL Analytics, Real Time stream processing, Graph processing, Machine Learning,  support for several programming languages  and many more.  " What happens under the hood, when we execute a Spark application? ".    thinking????? This is a very common question, which crops up in every Apache Spark user's mind. If you have the same question, I 'll answer this using Spark SQL by executing one of the most common data processing operation i.e. "Join". Spark being a vast technology, it is nearly impossible for anyone to cover all of Spark's details in a single article.  However, the details I'll cover here, should be enough to give you an idea of what goes on behind the scenes.  Prerequisite - This article ...
Read more

Spark Optimization - Bucketing

By
Image
Complex data computation is a part of every Big Data Project.  Do you have a business scenario requiring complex join on data having billions or trillions of values?  If you are using Apache Spark, stay tuned for next few minutes, as this article will address this expensive join problem, using Spark's optimization technique called Bucketing. Before reading this article, I recommend you to once go through my previous articles on Spark's Sort Merge Join and Broadcast Hash Join . They will set up your foundation for understanding bucketing technique. It is recommended but is not mandatory, as you would still be able to grasp the core concept. When to use Bucketing ?    For any business problem, where we need to perform a join on tables with very high cardinality on join column( I repeat very high ) in say millions, billions or even trillions and also this join needs to happen multiple times. Lets understand bucketing using the data of a hypothetical service ...
Read more

Apache Spark - Explicit Broadcasting Optimization

By
Image
Join operation is an indispensable part of majority of Big Data Projects. If you are using Apache Spark for your Big Data processing and using join operation, in this article I'll explain a strategy to optimize joins. The technique I'll explain is how to explicitly broadcast a table, in any Spark join operation and convert the join operation to Broadcast Hash Join. Why Broadcasting - W hen we perform a join operation in Spark SQL, b roadcasting proves very efficient in reducing data shuffling and hence serialization and deserialization of data over the network, which happens during a Sort Merge Join operation. If you haven't read my previous article on Spark's Broadcast Hash join, I highly recommend you to first go through that. This would provide you most of the deeper details of Spark's Broadcast hash Join. You can please find it here :- Broadcast Hash Join - By Akshay Agarwal Business use case  -  To explain this optimization, let's take the business...
Read more

Fraud Detection - Using Graph Database

By
Image
Graph DBs have become a buzzword in recent years.  In addition to social networks and recommendation systems, fraud detection is a field where the value of graph technology cannot be understated.  In this article we'll learn how Graph technology can be used to detect fraud.  If you are new to Graph technology, and haven't read my previous publication Graph technology in plain english , I highly recommend to read that once. It will help you to understand the fundamentals of Graph technology and serve as a catalyst for your learning process.  Fraud detection According to the research published by Crowe in July 2019, global fraud losses equate to shocking US$5.127 trillion each year . Fraudsters commit monetary frauds almost every day.  What if such fraud can be detected at an early stage, through prior experience of  some common fraud patterns  or known fraudsters.  We'll look at how Graph can be used to identify one such link and how tedious it...
Read more

Apache Spark - Broadcast Hash Join

By
Image
In this technical article, we'll discuss about one of Apache Spark's efficient join technique called Broadcast Hash Join. I will explain in detail how this operation is implemented internally in Spark at DAG(Directed Acyclic Graph) level and also different phases of DAG. We'll also uncover the details of division of spark application into jobs and stages.  Data  -  To explain this join concept, I'll take customer data of a hypothetical online service provider organization with international presence. Different customers across the world using its services holds accounts with this company. Moreover the customers uses different devices to access this company's website. Hence to fetch the details of different  customers(business_accounts) and the corresponding device used by them, we'll use two datasets: Business Accounts data   -  Data consisting of details of the customer's business account, created with this service providing organizatio...
Read more