Download System Design One Shot Full Course Subtitles & Captions | Learn Architecture

Welcome back aliens. My name is Vin

Reddy, and in this video we'll talk

about system design. I know you are

waiting for a long time for a system

design course on Telesco, and finally we

are here. Now, in this course we are

going to understand different aspect of

system design. Now, why it is important?

When you go for the interviews, most of

the companies will ask you for system

design knowledge, and the question will

be based on that. Now, of course they

will not be saying this is a system

design round, but they will expect from

you to understand different concepts

here, and how will you solve a complex

problem. And let me give you a statement

which highlights this, and I have it

here. Anyone can write a code that

works. A system design is what makes it

work for million people at once, and

that's the power of system design. See,

when you write a code, of course when

you learn a language, you learn a

framework, you can build application.

That framework can I mean, the entire

project will work. The moment you deploy

it, the moment it is in production, and

when you have users, not just 10 users,

of course it will work for 10 users, and

it should work for 10 users, but the

moment you have 10 million users

accessing that at the same time, will it

work? That's the question. When you talk

about all your favorite applications, it

can be WhatsApp, YouTube, Instagram, or

Amazon for that matter, there are not

few users who are accessing that at one

time. There are millions of users, but

still if you can see, they work. So, how

do we do it? And that's where system

design plays a huge role. Now, to

understand this, we have we will be

going through these concepts. So, in

this particular video, we are going to

start with the basics of system design,

what is system design, what are

different components, data versus

compute intensive, and functional and

non-functional, and all this comes under

foundation. Now, once that is done,

we'll talk about how the communication

happens between different systems. So,

we'll focus on DNS, API, and of course

REST, which you might be knowing if you

have worked with Spring Boot or Fast or

Django and any web framework. Then we'll

move towards the data. Of course, when

you have a lot of users, how will you

store data? Not just their personal

data, but their activities. Uh and in

that we have different options like SQL

databases, no SQL databases, when to use

cache, and when to use SQL and no

non-SQL. Then, let's say your users are

growing day by day. How are you are

going to manage your application? And

that's where scaling and distribution

comes into picture.

In which we'll talk about load

balancers, replication, partitioning,

and the CAP theorem. That this will be

an interesting the CAP theorem. And

then, we'll talk about operations and

reliability. And the most important part

there becomes a fault-tolerant. What if

something goes wrong? Does the

application do have something to do

that? In which we'll talk about message

queues and monitoring as well. And of

course, we'll cover different examples

here to understand different concepts.

And the entire section or this part will

be taken by an amazing trainer, Akshay,

who is very good in explaining stuff.

And I hope you will enjoy the entire

part. And do share your thoughts in the

comment section. Don't keep it blank.

You what you like, what you don't like.

If you have any questions, do point it

there. And we'll try to answer them. And

if you enjoy this type of videos from

us, let me know in the comments as well.

That is our motivation. And I hope

you're motivated. Let's get started.

Akshay is all set.

>> [music]

>> Hello everyone. Welcome back to

Teleschool. My name is Akshay Agarwal.

And today, we are going to start the

system design course. In this course, we

are covering each and every topic which

includes in system design. So, let's

start with understanding why do we need

system design at all? In systems, we

have a lot of systems around us. If we

talk about human body only, we have

digestive system. And also, we have a

system. Along with this we have a lot of

systems, but these kind of systems are

pre-built. It means we have not created

them. We just discovered them with time

and we got to know about various

components which are required for us to

breathe or to digest our food. So, in

these systems as they are not built by

us, we can't actually change anything.

Maybe through surgery we can change some

of the other component, but not much.

But the kind of system we talk about in

IT industry are the systems which are

built by human or engineers like us

only.

So, in that there is a possibility that

we can pick our systems based on our

requirement and also based on our

requirement we can pick the components

which are required in our system and we

can basically improve our system based

on the request we are getting, based on

the load our system is managing, based

on the kind of traffic which we are

getting in our system. So, in system

design if we go through the words we

have two words here, right? One is

system and one is design. What is a

system? If we go through the definition,

system basically means there are a lot

of components. There can be more than

one components in that system. So, let

me write components. And those

components are working eventually to

solve a particular use case or maybe

multiple use cases, but the common

purpose of the application remains same.

So, that means if we write system is

equal to components plus common goal,

this sums up our definition of system.

Now, when we talk about design, there

are various components which we are

going to learn and there are various

common goals based on the requirement

analysis of different applications which

we are going to cover in this particular

series. But in order to start, let's not

dig too much into the technical details

in this video. Let's start from a very

basic. Let's say we have a requirement

of creating our own application or our

own bank. Let's say we have a bank named

Alien Bank because we are in Telesco,

why not? And in this bank, we are not

dealing about each and every operation.

Currently, we have a single operation,

which is we have a cash counter, and we

have a person sitting on a cash counter.

Let's name it as cashier. Okay, so here

we are ready with our cashier, who is on

our cash counter, and the requirement or

the flow is pretty simple. For the flow,

let's understand the concept of our

bank. So, we have multiple, let's say,

customers are here, and each customer

will come to our counter, either

withdraw or deposit some amount, get the

receipt from the cashier, and then move

forward. So, let's write it down as

customer flow cycle. So, in our customer

flow cycle, there are four steps,

basically. Customer first will visit the

counter, will come to the counter, then

the customer will either withdraw or

deposit some amount, take the receipt

from the cashier, and then the

transaction is done. The purpose of our

application is done. So, these are the

core functionality of our system. It is

very basic, very easy. You might have

seen this already in some of the banks.

So, we are building the same

architecture right now. The architecture

basically looks like this. Now, in order

to understand system design, what it

really is, and how we are going to

improve the working of our Alien Bank,

let's face some issues. We are going to

face five issues, and based on the

issues and solutions, we are going to

understand how system design looks like.

So, here comes our first issue, which is

the process is really slow. By slow,

what we meant is, let's say, our cashier

takes 10 minutes per customer. So, that

means we are serving only six customers

in 1-hour span. So, this is a huge

issue, because if the number of

customers will increase to a limit, and

also we have rush hour, in those cases,

our system will break, and we will be

not able to serve the customers rightly.

So, in order to improve this, let's

first focus on our cashier. So, let's

understand our system from the cashier's

point of view. So, basically, our

cashier is taking time to understand the

requirement. By requirement, I mean

whether the customer needs to deposit

the amount or customer needs to withdraw

the amount. They have to make this

interaction, which is time-consuming,

right? Now, once the requirement is

locked, whether the customer is going to

deposit the amount or withdraw the

amount, the cashier manually have to

count the cash. And then our cashier

will prepare a receipt, which will be

again given to our customer. So, these

are the time-consuming tasks for our

cashier. How can we improve the cashier

time? So, in order to improve the

working of our cashier, we have three

major issues. First is they are not able

to understand the requirement. Next is

they are counting the cash manually,

which is taking time. And again, they

have to prepare the receipt. So, these

are the issues which our cashier is

feeling, and we are trying to solve

these issues. Maybe we can train our

cashier in order to count the cash very

rapidly. Okay. Also, our cashier is also

responsible for maintaining the receipt.

So, we will improve the typing speed of

our cashier in order to make the receipt

very fast. So, the second thing is train

to type fast. So, with this, what we are

doing, we are not adding any resources

to our cashier. Simply, we are making

our cashier super fast, right? We are

improving the skills of our cashier,

which is required for them to maintain

this job. And let's say through this

training, the cashier is really improved

now.

He is or he or she is now taking, let's

say, 5 minutes in order to fulfill one

customer's need. So, this is a huge

improvement. By this, we are able to

improve the process by 50%. So, let's

say our second issue is that the

customers which our cashier used to

attend are increasing very fast, because

this is our alien bank, and it is

working very fastly, right? Growing very

fastly. So, now the issue two is our

customers count is increasing. This is

our issue two. In this case, what we

know so far? We know that our cashier we

can't improve our cashier performance

because it is now at its peak. So we

have to basically handle some or the

other components to our cashier in order

to make the work very fluently. So maybe

the desk on which our cashier is

sitting, we will increase the size of

desk. So next thing which we can do is

we will provide the cash counting

machine to our cashier so that the

manual cash counting time will get

reduced, right?

So we'll provide cash counting machine,

right? So with this also the time will

get reduced and our cashier will be able

to attend more customers. One more thing

we can do that is instead of coming to

counter and getting the requirement at

the real time, we can give a form in

advance to all customers so that they

can fill it and our interaction time

will get reduced, right? So our cashier

will now don't have to communicate each

and every request to our customers.

Uh they can simply see the form and

based on the form they will fill out the

thing. And now our cashier have cash

counting device also, so the process

will be fast. Let's say through these

processes, through enhancing our desk or

cash counter, the timing got reduced

from 5 minutes per customer to 3 minutes

per customer. This is also a huge

impactful thing. So now our cash counter

is capable of attending more customers

in lesser time. So that means even if

the customer count is increasing, we

will be able to process more customers

and enroll more customers into our Alien

bank. So this issue is also kind of

solved. Now let's move to the third

issue. The issue three is our customers

are still waiting to get their

application resolved, right? Currently,

what do we have? We have our cashier who

is upgraded to its maximum level. So, we

can't upgrade the cashier more. Also,

the desk on which it is sitting, the

cash counter, is also upgraded to its

maximum limit. So, we can't upgrade our

cash counter more. So, these two things

are pretty solid right now. We can't

improve it now. Now, the thing is, so if

we have 10 customers in place, that

means our 10th customer will have to

wait for 27 minutes in order to process

the three-minute work. So, that is a

huge drawback of our system right now.

And why that drawback is coming? Because

we have only single cash counter in our

system. So, in order to make the process

more feasible, now we have reached that

point that we have to add more cash

counters into our system.

So, let's add them. So, now we have our

cash counter one and cash counter two.

Now, at this point, we have added one

more counter, and things are pretty

solid right now. So, our customers will

eventually have two counters to go to,

and both of those counters will be able

to solve our issues. It feels good,

right? Now, we have two counters to

process. So, our customers will not have

to wait for longer time. They can either

go to counter one, or they can go to

counter two. But, let's say if first

customer is going to counter one, and

the counter one is able to process the

information, and they are maintaining

their own data. And similarly, our

second counter is also maintaining its

own data. And they are not synced. So,

this creates two issue, right? First

issue is, if in our system, the counter

two doesn't know anything about the

customer, that will create an issue

because they have a valid account in our

bank, and they should be able to proceed

their information through it. Now, the

second issue is this only. That from the

first account, they can also withdraw

the amount, and they will not update the

second counter. So, that means they can

again withdraw the amount from cash

counter two also. This will create an

issue of data discrepancy. That means

our data is not synced. So, the fourth

issue here is we need to have a

centralized database or central data

from which the cash counter one and cash

counter two will go and validate the

data and go and retrieve the data. If

the account is there for any customer

and the amount is also there, so the

customer one will also insert the cash

into the same account. And if the

counter two is getting withdrawal

request, they will also get the amount

out of the same account. So, in order to

fix this, we need to have a centralized

data. That means our counter will look

like this. This is our counter, let's

say two. This is our counter one. And

they will eventually connect to a

centralized database. And as it was

before, our customers are standing in a

queue waiting for their turn to come in

order to get the request. Now, after

solving all these case also, we have a

new issue in hand. Which is, let's say

the gate of our bank is here. This is

our alien bank. So, since the gate is

here, most of our customers are going to

the cash counter one only, where cash

counter two is sitting idle. Or maybe

processing lesser request, far lesser

request than the cash counter one. This

will again end up adding the wait time

to our customers. And again, we will be

end up handling lesser customers than

our potential. So, let's write that

down. This is our issue five, which is

underutilized counter. Now, in order to

solve this issue, we need to have a

middleman. Which will stand apart from

our counters. Let's say this is our

counter two. Again, this is our counter

one. And here are our customers. And now

our customers will not go to the counter

directly. They will first go to this

middleman. Then the middleman will check

how much load each counter is handling.

Suppose there are 10 customers at

counter one and only five customers at

counter two, then our middleman can

decide that next customer should go to

the counter two.

And the count will increase to six,

right? And the next customer will again

go to counter two, and the next customer

is again going to counter two only.

When the limit reach, let's say we have

11 customers here and 11 customers here.

And if then any new customer arrive,

they will again ask man where to go.

Let's say on some fine day our counter

one is not working. That means our

middleman will know about this and will

eventually send all the customers to

counter two until the counter one is

getting fixed. So, let's end this

example here at this particular point

and see what have we learned. Through

this story, through this single story

and five issues, I have tried to capture

multiple concepts of system design. So,

the first concept which we discussed is

designing a system, that is for sure.

Ultimately, what design we do have here?

We have one cashier and the other

cashier. Each cashier is having their

own counter. So, this is our counter

two, maybe this is our counter one. We

have a number of customers waiting for

us. Also, we have a middleman now. With

that, we also have a common database,

okay? So, this database is nothing but

centralized database. The customers

which are coming to our bank is nothing

but requests. And if they are standing

in a queue, we can say they are creating

a traffic, okay? So, these are the

traffic which we are going to handle.

Now, all the requests or all the traffic

is coming through a load balancer, the

middleman. So, this is nothing but our

load balancer. So, from the load

balancers, all the requests are coming

to either of the cash counter. So, we

can say that our cash counter is nothing

but servers. And the person sitting

there doing the work is nothing but our

application code. Now, let's see the

system again through the system design

point of view. So, the first issue which

we have was our process is slow. So, in

order to solve that, we improved our

code quality, right? So, first thing

which we do in our systems also, we

improve our code quality. So, this is

where our LLD will come and this is

where our DSA will come. Suppose a

particular loop is taking certain time,

we can reduce that loop or we can change

that loop to some for each function or

some other way in order to process that

information fast. So, while solving the

issue two, we increase the size of our

table, we added some functionality, we

added this cash counter. That means we

upgraded our server. So, in order to

upgrade our server, this term is called

vertical scaling. In vertical scaling,

we don't add multiple servers, we

basically update our current server

only. And again, the current server

updation have a particular limit. After

that limit, you can't upgrade your

system. So, then comes the issue third,

which is our customers need to wait for

the single server to respond. So, in

this case, we added more servers. So, in

this scaling, we added more server that

is called horizontal scaling. So, after

having multiple servers, we need to

maintain the data at the common point.

So, that's why we added databases. And

in order to add a common database, we

basically can move to distributed

database architecture or can move to a

centralized approach of handling that

database. So, this is nothing but

centralized database. So, after having

centralized database, again the issue

comes where we don't know how to handle

multiple requests. That means our

requests are again going to one counter

only or one server only. Now that we

know they are going to one server only

and the other server is underutilized.

So, what we did, we added a middleman.

Our middleman is no one but our load

balancer. Our load balancer basically do

two things. They will check whether our

servers are healthy or not and based on

that, they will distribute the request

equally into multiple servers. So, these

are all the concepts which we discovered

through this particular example. So, if

we think about the applications we are

using in our day-to-day life, we are

using WhatsApp, we are using maybe

Snapchat, we are using Instagram, we are

using YouTube, we are using LinkedIn, we

are using Netflix, Amazon, maybe

Flipkart. All of the applications, if we

think about it, are showing us the data

either in the form of images or in

videos, in audios or in text. So, we can

say through a large lens, the data

consist of these four kind of

information. And all of the applications

which we are using in our day-to-day

life consist of these data's only. So,

we can say that in the core of every

system, we have data and we are then

finding ways to make it available to the

users in the way they are expecting it.

So, in the center, as we discussed, we

have data. Data can be of various form.

It can be image, video, audio or text.

Now, if we think about data, it is not

like we are seeing the data once only.

We have to see the data and also while

storing also, we need to store the large

amount of data. Now, in order to store

the data, we do have databases. So,

databases are basically our first

component of system design. What is

functionality of databases? Pretty

simple. We need to store our data and

whenever we want to retrieve some data,

we need to get the data from our

databases only. They are represented by

this cylindrical figure. Databases can

be of multiple ways. It can be

categorized into no sequel or sequel.

Now, there are various kind of databases

in both of these categories. In sequel

also, we have various databases from

which we can select which database we

are going to use. And similarly, in no

sequel also, we have various kinds of

database from which we can pick which

database we need for our application

right now. But, the core remains seems

that we are saving our data in some or

the other database. We will discuss

about sequel and no sequel databases in

the coming videos. That's why I'm not

going to deep into this at this moment.

Let's think about the scenario where you

and every user on the planet knows the

queries of database, knows how the

tables are structured if we are talking

about sequel, or how the documents or

nodes are connected if we are talking

about no sequel. If we talk about both

of these scenarios, if user, the end

user, already know the database query

languages and know how to deal with

different kind of data, in that case, we

might not need any application if you

think about it. We will simply go to our

database, or the database will be

accessible throughout every application.

Suppose we are seeing the LinkedIn, but

on LinkedIn we will see an interface on

which we will write our query, and the

data will be fetched to us. That will be

an hectic task. But, if users know about

that, and if users start exploring that,

that will be easier, right? In that

case, we don't need any application

layer in between us, and in that case,

we don't need any client-side

application also between us. We simply

will see a database, query it, and get

the results, right? In that case, what

will happen? Each query have to be

managed through the users only. And

these queries, when I talk about large

systems, let's say we are talking about

LinkedIn, there can be multiple tables

through which our data is curating and

coming to us. Every user can't do it,

right? That's why we have different

layer in front of our database, which is

our application code. So, in application

code, what we do is we hide the

complexity of dealing with databases,

different tables, different documents,

and we give the endpoints to our

application or to the world so that they

can access the kind of information they

are looking for. For that, we create

APIs or we can say we create endpoints.

There can be multiple endpoints based on

different features, based on different

operations. So, through endpoints, we

communicate to our application, which

will again communicate to our databases

to get the data we are trying to get.

There can be various operations, like we

can insert some data, we can update some

data, we can retrieve some data, we can

delete some data based on our request.

So, the thing becomes little easier to

us. Now, we don't have to deal with all

the database tables, but now we have

assigned endpoints through which we can

do the work which we want to do. And

endpoints are pretty simple.

So, in order to get some data, we have a

get endpoint. Let's say https, and here

comes our Teleschool. So, this URL or

this endpoint can be used to fetch all

the students which are there in our

Teleschool environment. But then, what

we are doing? By introducing application

code, we are moving our logic and data

separately. That means we need some kind

of connections between these two. In

which we can connect through IP

addresses, we can connect through some

HTTP protocols, or we can connect

through some TCP protocol. There are

many ways of connecting the database

from the application code and getting

the data which we want. So, in order to

show the APIs or in order to interact

with the APIs, we have to know what is

JSON and how we are going to send the

data to our APIs and how we are going to

fetch the data from our API.

In this level also, if we know how to

handle JSON, if we know how to handle

various operations through which

endpoint, it will make our life little

easier than accessing the databases,

right? So, in that case, we don't need a

front end if we know how to hit the

endpoints through which we'll get the

data. But, for our normal users, for

other users of any applications, this is

also a tedious task. So, in order to

make this easier, we have a different

layer between our system that is our

client application. Client applications

are nothing but the applications through

which we are interacting to the

database, through which we are getting

those databases. So, our client

applications, be it a mobile

application, be it any web application,

be it any ATM machine for that matter,

they are connected with the application

code, and further, the data is being

consumed by us. So, again, in order to

connect the application layer and the

client code, we again need some kind of

communication between them. We either

can go to the same HTTP or TCP or maybe

IP. Now, we have our application ready

in our hand and we can use it. So, if I

draw it like we used to draw it, this is

represented by our client, either a

desktop or a mobile device. Now, they

are sending queries to our application

code, and our application code is

further getting the data from database.

Now, for our client to communicate with

our application code, we need to put our

application code on any server. And now,

if we think about it, our application

code is already exposing the APIs to our

client. That means, it can also expose

the APIs, or any other application can

also use our application codes through

our APIs and do the work which is

needed. So, it can be connected to any

other server also, which will contain

their application code. And this will

communicate with our data through our

application code. For every query which

we are making, we are going to our

database, and from that we are getting

the data which we are trying to get. So,

if we think about it, most of the

communications will start from the

client side, and they will come to our

application code through which we will

get the data from the database. This is

pretty much the easy application or the

applications which we create on our

college projects look like. Either a

to-do application or a simple chatting

application will look like this. Now, in

the current scenario, what we are doing

with this kind of architecture is every

client is going to ask application code

for some or the other information, and

to fetch the data, our application code

needs to get the data from the database

only. We don't have any other option. In

order to make the files accessible to

our server faster, because the

database's calls are heavy, which will

take a lot of time, let's say, if a

database query is finding the columns

from 12 tables, it is going to take

time, right? Now, in order to make the

frequently used data accessible faster,

we have a different layer, which is

called cache. When we are going to put

any data into our cache, and when we are

supposed to remove the data. There are

different strategies we are going to

cover that also in the upcoming videos,

but for now, let's see, this is our

cache, and all the frequently used data

will be stored here. So, that we don't

have to call to our database, and they

can be easily accessible through our

cache only, which will make our process

slightly faster. So, this is our fourth

component of our system, which is cache.

Now, let's move forward. Let's say our

application is big, is having multiple

servers. In that case, we may have some

client or this client, either the mobile

application or the web view, and we are

getting the data from multiple servers.

Let's say this is server one or this is

server two. Now, when we have two

servers or multiple servers, our client

have to decide where to go. From where

do we fetch the information? Now, in

order to take this control, we have a

load balancer in the middle, which will

take care of each and every request and

which will take care of each and every

connections of the server. So, our load

balancers are responsible for handling

two tasks. First, they will check

whether the server is healthy or not.

And second one, they will divide the

request for each servers equally. This

is what our fifth component looks like.

Now, let's come to our next component,

which is messaging queue. So, in order

to understand message queue, let's first

understand that in order to process one

operation, there can be multiple task

into that operation, right? Let's say we

are ordering something. So, when we are

ordering, we are going to do multiple

tasks, right? We are first creating an

inventory update. Next is we are going

to process the order also. We are going

to talk to the vendor who is going to

take the delivery. So, we are going to

handle delivery. And then, we are also

trying to send you the information, the

notifications that your order is placed

successfully. Don't worry about it. So,

that means we are going to handle the

mail service or the SMS service also.

Now, in order to handle multiple

services like this, we can't wait for

every application to respond back. There

are two kind of applications basically.

First is, let's say this is an

application and this is another

application. Let's say this is SMS

application and this is our order

application or microservice for that

matter. Now, whenever an order is

placed, it is going to speak to the SMS

service or SMS application to send the

message to the user that your order is

placed successfully. Now, there can be

two ways of handling this. First way is,

whenever the SMS is sent, they will

notify the order service or the order

application back that we have sent your

message done. Or there can be different

way of handling it that from our order

service, we simply say that okay, this

is our order information. Please send

the message to this number and we are

done. We will not wait for the response.

We will move forward with our

application and with different orders.

So, the kind of requests in which we are

waiting for any confirmation or any

feedback from the other service are

called synchronous calls. And the kind

of service in which we will simply put

down the service and we are not waiting

for any confirmation, we will move

forward to do our task. Those are called

asynchronous services. Don't worry about

this if you don't understand it now. We

are going to dissect each and every

component in details in the further

coming videos. So, for asynchronous kind

of services, what we can do, we can

simply push our notification or push our

request to another service and move

forward. So, in those kinds of services,

we have a message queue or message

broker in middle. What it will do, we

are having our order service here. It

will put multiple request or can put a

request time to time into this message

queue. And then all the requests which

our order is processing through message

queue, they will not take care of it

anymore. Our message queues will take

care of it. They will connect with our

SMS service or SMS applications and send

the request one by one. And they will

take care of these all operations. So,

our message queue is like a broker who

will take the request from one

application, store inside it. They will

take care of these applications. They

will take care of the retries or if any

request is failing, they will take care

of it. Maybe they will communicate to

our order application in the end or in

the end of each day, we will send some

notification, some batch processes to

our order application in order to show

that which requests are fulfilled and

which are not fulfilled. So, in short,

our messages queues are doing two work

here. They are storing each and every

request which are coming from various

other services or various other

applications for that matter. They are

storing it and they are taking care of

those requests. They are sending those

requests finally to SMS application or

service and they are taking charge that

okay, my first service or my first

request is being processed. Nice. Now, I

will send the next request. Now, I will

send the next request and further on. If

it is failed, there are various

mechanisms how messages queues work and

how they bisect and dissect each and

every request and make sure that they

will provide the latest status of each

request. And in those kinds of

environment where we have multiple

microservices, in those cases message

queues plays a vital role in between

them. And with that, let's move to our

final component which is monitoring and

logs. In monitoring and logs, what do we

do? As we know now, we have different

components in our architecture, we have

a client system here.

We may have a mobile application again

which is connecting through an API to

our load balancer

which is again further connected to

either server one or server two.

Our servers are the area in which we

have our application code and then they

may be connected to single database or

multiple databases and those databases

can also communicate with each other in

order to maintain the data. And also,

let's say our application is also using

some third-party application for which

we are using message queues. Our message

queues are denoted by this. Now, this is

our architecture. We have so many

components.

So, it is very important for us to

monitor those components. To monitor

whether our load balancer is working

finely or not. To manage whether our

application code does have any error or

not. And if there is an error, we need

to know how to solve it. We need to know

the complete stack trace of that

particular error. We need to know how to

reproduce those errors on our local

system in order to fix them. And also,

while working with the applications, we

know that for every each and now, we

have multiple features which are being

attached to our applications. So, we

also need to monitor while adding one

more feature to our application, whether

any other feature is impacted or not, or

it is failing or not. So, in order to do

that, we have a lot of monitoring tools

and also logs to check our system. There

is a complete testing department in our

team also, who takes care of these

things. This is one of the vital role in

our application. So, these are most of

the common components which are used in

system design. Whenever we think about

any application or designing any

application, we suddenly jump to the

tools which are used in that

application. We start creating diagrams

that this is our client, this is our

database, this is how our data is going

to flow, this is how we add cache maybe.

So, these all things we start doing

without noticing a single thing, that

is, how is our application going to

work?

What is this application? What are the

limitations of this application? In

system design, there is a smart way of

doing that. First question you need to

ask yourself before diving into the

solutions is, what is this application?

What is the type of this application? Is

it a data intensive application or a

compute intensive application? With this

common differentiation, you will be able

to save a lot of cost to your system

design, as well as you will be able to

handle the task properly into your

system. Let's say, we have two

applications. In both of these

application, there are the same number

of users. With that, the network on

which both of the applications are

working are also pretty much same. Now,

let's put a scenario that our system or

the application which we are working on

is facing latency. Now, what is latency?

Latency means our system is taking time

to respond. There are various ways of

handling this issue, of fixing the

latency issue. We can add multiple

databases, we can add different caching

layers, we can update the caching

mechanism, or we can update

or the CPU powers. Now, on which vector

we are going to do this changes defines

how well we are going to solve this

issue. So, now there are two type of

applications as I discussed. One is data

intensive application and one is compute

intensive application. Let's understand

them one by one. The basic difference on

which data intensive application is

different from a compute intensive

application is that in data intensive

application our core focus is on data.

Now, for this data we need to have

multiple operations, right? We need to

store the data, plus we need to gather

the data, we need to move the data time

to time. That means we need to keep

updating the data because new data is

entering. And also, while new data is

entering we can also, right, we need to

add a lot of data.

We can take example of any application

which can be Instagram because on

Instagram we see a lot of content and we

post a lot of content. So, that is a

data intensive application where the

primary focus is on gathering the data,

is on showing the data, is on the data

you are posting. But, the focus is not

on how the data will be calculated. The

calculation part is pretty solid. We

just need to move the data from our

databases to the client applications and

we are sorted. So, in these kind of

systems the data mainly resides in the

database. So, first issue can be the

database response is low. Another is,

since we are using network calls here

and there, that means our network calls

are also bulky or we need to optimize

them. The third can be our server itself

that our server configuration is pretty

low, that's why we are not able to get

the data faster. So, all in all, in

data-intensive applications, we are not

blaming the client to upgrade their CPU,

we are blaming our circle, our database

calls, our network calls, our server

configuration in order to get the data

fast. Most of the applications in which

you work on are data-intensive

application. That's why we use terms

like cache enhancement, shredding, and

through all of those things we will be

able to solve this issue. Now, let me

also write some common examples of this.

So, the common examples here are

Instagram feed where we see a lot of

data as we discussed earlier. In

WhatsApp messages also, there is a high

amount of data because we are not only

taking care of one chat, we are taking

care of entire chat. The count of chats

or the count of messages WhatsApp deal

with is around 2 to 10 million messages

per day. So, this is also a heavy

data-intensive application. Now, there

are bank transactions, analytic

dashboard, and log process system as

well. In all these application, the

primary goal is to serve the data from

database to our client system faster.

There is not much calculation required

for them to show on the client machine.

Now, based on the data-intensive

application, what are the worries of a

data-intensive application? The first

worry is how fast can we read the data?

The second point is how safely are we

going to store the data? The third

point, as we all know, a lot of users

are using our application, so how many

users can use the same data

simultaneously? The next and one of the

most important worry of these kind of

application is what if any server dies?

What if any server breaks? What if any

network call breaks? What if any

database machine breaks down? So, in

those cases, how we are going to handle

the data? So, these are the most common

worries of the data intensive

applications. And now, in order to solve

them, we don't go to our client system.

We talk about the components on which

these processes lies. And those are our

databases, caching, replication, maybe

data sharding, or it can be solved

through data consistency. So, these are

all the components or the terms we use

in order to improve our data intensive

application. Don't worry about these

terms if you don't know them yet. We are

going to cover each and every component

in details in the coming videos. So, I

hope data intensive applications are

pretty clear up to now. But, for the

sake of example, we are also going to

take one more example. Now, think about

Instagram feed. We are not solving

Instagram feed issue through any

mathematical equations. We just have to

gather a lot of data and show you that

data. So, what we don't do here, we

don't do any complex maths. What we do

then? We normally fetch the posts. After

fetching the post, we sort them through

time or relevance. We show a lot of

images or videos. And according to those

content, we normally do some operations,

maybe like or commenting or sharing. So,

I'm writing that. So, this is all the

operations loosely we have on the

Instagram feed. Now, what is the common

challenge which Instagram is facing

daily? That is we we have a lot of

content. We have millions of users and

million of content which we have to

handle. So, let me write that down. In

other words, we can also say that

million of users are reading and writing

at the same time. So, with this

challenge, Instagram still needs to

scale, right? How does Instagram scale?

If we think about this, Instagram adds

multiple databases in order to store the

data. And also, Instagram uses

aggressive caching in order to store or

get the data fast. Now, we also know

that Instagram heavily depends on the

images and the video content form. So,

in order to handle them, we can't simply

store those kind of data into our

databases directly. For that, in order

to fetch them faster, we need to have

certain CDN in place. And in databases,

we will only store the location of that

particular file which we are going to

fetch through the CDN. Now, think about

this. In order to fix the latency issue

on Instagram or in order to help

Instagram scale, we hardly touch the CPU

or we hardly touch any GPU. We only

touch the data or the components which

are associated with them. Because the

issue here is not the CPU, but the data

movements between the components. Now,

this is all about data-intensive

applications. Let's now focus on the

compute-intensive applications. Now, the

compute-intensive application is exactly

opposite of a data-intensive

application. Here, the amount of data

which we are retrieving from databases

are pretty much low, but the

calculations we need to perform are

really heavy. So, instead of putting our

focus on database caching, replication,

sharding maybe, we move our focus from

there to CPU and GPUs. So, let me write

down the challenge now. So, we can

safely say that here our systems are

compute-bound or we can say CPU or

GPU-bound. Let me take some examples of

this. The most common example can be

image processing, video rendering. The

machine learning model training can also

be a good example of this. Also, we can

talk about simulation and cryptography.

Now, in all these examples, the data

which we are getting from the database

are in a smaller level only, but the

kind of processing we need to do in

order to show these data to user is

heavy. That's why they come under the

category of compute intensive

applications. Now, let's discuss about

the worries of a computer intensive

application. The first worry is pretty

simple, that is how fast can we do the

calculation or how fast can we compute?

The second worry can be can we do

something some processes in parallel in

order to show the data faster to the

user. You may have seen a lot of systems

in which in parallel the image is being

processed and you are able to move

further with the application. Let the

image process and the meanwhile you are

moving to the other features of the

application. The next issue can be how

can we reduce the computational cost?

And also as a system designer, we can

also think about can we use GPU instead

of CPU? So, these are the common worries

of a computer intensive application

where the data is not much, but the

processing is the king. Let me also take

a good example of a computer intensive

application. So, in order to create a

simulator machine, it is very important

for the simulator to work exactly how

the actual machine works. So, think

about a scenario that our country is

hiring a lot of pilots every year and we

can't spend too much on the actual

aircrafts when the pilot doesn't know

how to ride it or how to fly it.

In order to train them, we need to give

them simulators machines and on which

they train themself first and then sit

on the actual plane. So, now in

simulation machine, our common worries

or the task which we are going to have

is can we do anything in parallel again?

Can we use GPU instead of CPU? Again,

how can we reduce the cost of our

simulator? Also, we need to find a way

to gather a lot of good algorithms in

order to do the task frequently. Now, at

this point I think we are very clear on

how our data intensive applications are

different from computer intensive

applications and why we need to put our

focus on computation on different

algorithms on GPU or CPU in order to

make a computer intensive application

works because the data is very low and

in data intensive application if we

start focusing on GPU and CPU there will

not be any big changes because the issue

lies in the database network calls

caching algorithms or maybe server. So

in order to fix them we need to put our

effort into the right category into the

right components so that our application

works seamlessly. And why this is

important? Because if we put our efforts

into the wrong components our components

are not free they are going to cost us

money and if we put the money in the

wrong place and not getting any help in

order to solve our issue it is going to

be a bad system design for the entire

game. Now when we talk about data

intensive or computer intensive

applications we often think that these

two are very separate applications. But

in today's time we live in the

environment of multiple applications and

each application performs multiple

tasks. So that means our data

intensiveness or the computer

intensiveness of the application can

also depend on the feature we are

working in. For example we have YouTube

right? YouTube is data intensive for

serving the videos. There are a lot of

videos in YouTube environment and you

get the videos and we can say that it is

computer intensive when it comes to the

recommendation. Because in order to get

the recommendations we have to get what

you are watching right now. So I think

it is pretty sorted now that in order to

fix a particular issue you need to first

think whether the issue which we are

facing is computer intensive or data

intensive and then you get into the

components and fix that. There is a

trick to understand whether the feature

is data intensive or compute intensive.

That is, if the time is lost in data

movement, probably it is data intensive

application. Or if the time is lost in

the computation, it is probably a

compute intensive feature or the task.

This one distinctions save you a lot of

money, time, and complexity. In order to

discuss our requirements, we divide the

requirements into two categories. One is

functional requirement and the another

one is non-functional requirement. So,

let's go to our screen and let's discuss

this. Functional requirement says,

"Please define all the points which you

are building, all the features which you

are building, all the working which you

are building." That means, if we define

every feature that comes under the

functional requirement. So, let's

understand functional requirement by

taking an example of any e-commerce

application, let's say Amazon. So, what

can be the functional requirements of

Amazon application? That means, we are

defining each and every feature which we

see on Amazon. So, let's say on Amazon

application, a user can register

themself. User should also be able to

log in. So, let's write that. After

login, user should be able to see the

products. Also, user can search the

products or filter them. Let's write the

searching and filtering into two parts.

So, search the products and user

can also filter the products. After

filtering, searching, user should be

able to add any product to the cart.

After adding any product to the cart,

user should be able to get some

promotions or get some discounts, right?

So, they should be able to apply the

coupon codes to get the discount. And

once all this done, user should be able

to place the order. After placing the

order, user should be able to do the

payments, maybe. And once the item is

placed, once the order is placed, user

should be able to track that order. So,

all of these points are the functional

requirements of Amazon or any e-commerce

application for that matter. In which we

define all the functionalities, all the

actions which user can take, all the

working of the application is defined

under functional requirement. So, if we

have to write this, what is functional

requirement again? Functional

requirements means all the features

which you are going to have in your

application are the functional

requirements. Now, let's move forward to

the non-functional requirements. For

non-functional requirements also, let's

start with the example of Amazon only.

Let's say the first non-functional

requirement is that on our Amazon site,

we have

approximately 1 million users. So, that

means our system should be able to

handle 1 million daily active users.

Now, in order to give our users a good

experience, we can say that the response

time should be less. But, in

non-functional requirements, we have to

define how less.

So, let's say for Amazon application,

the response time should be less than

200 milliseconds for each request. Now,

our application is built and is running,

that means our application can't handle

failure. But, that is not possible,

right? So, we should define up to how

much is our availability of our system.

That means for how much time our system

will be available. That are defined

under SLAs and SLOs. So, we have to

define them. That for 1 year, our system

downtime should be this much amount

only. And for 99.9%

time, our system should be up. That is

defined under availability. After

availability, we don't want our data to

be flowing easily, so we want to encrypt

our data. And let's say on any sale day,

let's say Black Friday sale or Diwali

sale or Holi sale, in those times our

normal users can go up to 10x. So, our

system should be able to handle that

much traffic also. So, let's write that

down. With this, our system should be

also able to handle the faults. Because

there are going to be faults in the

system. No application is perfect so

far. Every system have their own faults,

have their own downtime. But we need to

see how can we track them and how can we

fix them as fast as possible. So, that

is we need to make our system fault

tolerant. So, with this example of

Amazon, we get to know that our

functional requirement are the features

of the system that we can say what our

system is going to do. And with

non-functional requirements, we define

how our system is going to perform. So,

these are the specific non-functional

requirements for Amazon. If we have to

write it in a generic way, there are

some generic terms which we can write.

So, we can talk about scalability. Let's

say our current configuration of server

is reaching to a point where it is not

able to handle the kind of traffic it is

getting. So, now in order to scale, will

you scale it horizontally or vertically?

That will be taken care under

non-functional requirements. Apart from

that, as we have discussed, availability

of every system will be defined under

the non-functional. As we have already

discussed, that means our system uptime

will be calculated here.

Which is defined under SLAs and SLOs.

So, in these agreements, it is already

defined that, let's say we are using

Amazon server in order to deploy our

application, then Amazon will give us

some contract that we will be down up to

this time. And apart from that, you will

not see any issues. Now, in those cases,

if we see any issues, let's say for a

day, Amazon says that we will be down

maybe for 5 minutes maximum. But then,

if Amazon fails to keep that promise or

that agreement, the companies which are

using the Amazon service can sue them.

Because the Amazon services have given

us the SLA's in writing that the

services will not be down after certain

point. So, availability plays a huge

concern there. Next, we have

reliability. In reliability, we can say

that our system should be reliable. That

means there should be no data loss in

our system. After reliability, comes the

performance. Under performance, we are

going to define how our system is going

to work, how much latency can we feel in

the system. What is our P95? What is our

P90? What is our P99? These all things

are defined under performance. So, all

the words which I have said in this

particular video, we are going to cover

each and every topic in details in the

coming videos. So, don't worry about

this if you don't get it now. After

performance, we can jump to security. In

security, we can have features like

authentication, authorization, and

encryption. With security, we are also

dealing with a lot of code. That means

we can say our code should be in perfect

modules so that each code or each chunk

of code can use different APIs or

different modules in order to work in

efficient way. After this, we have last

but not the least requirement, that is

observability. Even if our code is

pushed to production, there are a lot of

use cases which are handled after that.

Because we don't know the exact behavior

of users, that how they are going to use

it. So, we need to keep track of all the

monitoring tools, all the tools which

are going to track the user behavior on

our application. So that we can improve

our system and also we will be able to

track if there are any bugs and solve

them. So, let me write monitoring and

logging. So, this is all about

functional and non-functional

requirements. I hope you all get it. But

in order to completely get it, we all go

through a lot of applications on a

day-to-day basis, right? We may be

scrolling through Instagram, maybe

watching YouTube, maybe watching

Netflix, or Amazon Prime, or purchasing

anything from Flipkart, Amazon, or any

other application e-commerce platform,

or we may be scrolling through our

WhatsApp messages. Now, I want you,

while using your application, just sit

back for a few minutes and think about

all the functional features which you

are working, or all the functional

requirements which you are using, and

all the non-functional requirements

which the app must have configured or

thought of while creating that

applications. Also, comment down the

application on which you worked on, and

write down three to four functional and

non-functional requirements. That will

help you and me also in order to

understand that we are learning on a

similar pace.

How does our browser know which webpage

to hit? Do they know by domain name, or

do they get these things through IP

addresses?

Let's find out. So, whenever on our

browser or on our machine, whenever we

write any address, let's say

telesco.com, it is going to redirect us

to the telesco page, which is our

homepage. Maybe some banners here, maybe

some courses section here.

It is like this only. Now, how do our

browser figure out which page to show?

Because, as humans, we communicate

through names, but systems don't

communicate through names. In order to

establish the communication between one

system to another system, we need IP

addresses. So, that means there should

be some place in between or some system

in between our browser or our OS through

which we are going to get the correct IP

addresses of each and every site. So, we

will send telesco.com, and in return we

will get some IP addresses. So, this can

be the IP address of telesco. I'm not

exactly sure about the IP address, but

yeah. Like this, our DNS works.

So, DNS basically stands for domain name

system. Domain means any specific URL or

any specific domain. Domain is this

only, telesco.com. There can be multiple

domains, telesco.net, google.com,

facebook.com, whatsapp.com, .web I

think. But yeah, you get the point.

These all things are domain. And what is

subdomain? If I write any subdomain,

let's say docs.telesco.com

or any other subdomain, let's say

course. So, they both are subdomains of

telesco.com. Where telesco.com acts as

an umbrella and under which we can

create multiple subdomains if and when

required. So, these are the domain and

subdomains. But mind it, the machines,

the computers doesn't communicate with

each other through these domain names.

So, they need IP addresses. Let's figure

out ways on which our system can handle

the IP addresses. Let's say we are

building a browser. Why do we need DNS?

Why do we need another system

for maintaining the domains? We can

basically install or store all the

domain name list on our browser only.

Now, is this even possible? Let's say if

we have only two or three websites,

maybe 10 websites, we can store their

names, even 100 also. But the current

number of domains which exist in our

world is more than 350 million. And as a

browser, browser doesn't know which

sites I'm going to use, whether I'm

going to use Stack Overflow one day and

Gmail the other. So, we can't store all

the 350 plus domains into our browser.

That will crash our system. So, two

reasons because of which we can't store

the entire domain list on our browser

are first thing is it will get heavier.

And to store such amount of information,

we need a high storage browser. So, that

will also impact the performance of the

system. Now, the other point is what if

the IP changes after some time? What if,

let's say, I am running any site,

teleschool.com, I have purchased my

domain from maybe GoDaddy, and tomorrow

I decide to discontinue my service from

GoDaddy, and I'll move to Google

Domains. So, I'll purchase or renew the

domain from there. In that case, my IP

can be changed. And if it restores, I

need to reassure that in all the

browsers where teleschool.com is

running, I need to update them. I need

to send them some update. So, through

this we can estimate that it can't be

stored in one browser. And also, it

can't be stored in a single server,

also, because it will also get heavier.

There are more than 350 million domains.

That means, if we are going to store it

in a tabular manner, so the here will be

the domain name, and the corresponding

IP addresses accordingly. So, here we

will have any domain name such as

google.com, and here we will have the IP

address, 1.2.3.4,

for example.

So, by this, we will create the entire

list. This is not possible for a single

machine to handle, because if that

machine breaks, then the entire system,

the entire internet will break. Anyone

in the world will not be able to use any

internet services or any web pages for

that matter. No APIs will hit, because

it all depends on the DNS. So, one thing

is sure, we need a certain kind of

system in order to fulfill this

requirement. That's why we have one of

the most beautifully designed system,

DNS. Let's start from the high level.

So, this is our machine, which is going

to use some internet service provider,

and under this service provider, we have

something called as DNS resolver. So,

this DNS resolver is responsible for us

to getting the correct IP of any

websites or any domain which we are

trying to process. In order to

understand this clearly, let me take

this and put it in a side, so that I can

draw more things onto our screen. So,

the first call it is going to make is to

our root server. Now, there are 13 root

servers

in this entire world. That means there

are 13 companies who own 13 web servers

in this entire world. They are not

physically 13, but they are categorized

into 13. So, that means out of these 13,

we are going to hit one of them and get

the response. These root servers are

typically named from A to M like this.

Again, from this A to M, they are owned

from various companies. So, you can see

the complete list of the root servers

with the companies which own that root

servers on the screen. You don't have to

remember all of them. You just have to

see and understand that these are the

service providers of the root servers

from which we are going to get some

information in order to process our

domain. Now, let's come back to screen.

As discussed, we were searching for

telesco.com. Also, I can remove this

now. Now, when we are searching for

telesco.com, the root server doesn't

know anything about telesco.com, but it

knows about the TLDs which are top level

domains. What is top level domain? If I

talk about .com,

this is our top level domain. .com,

.net, .gov, .in, .uk, .us, .education

maybe .edu, we use them. These all

domains are the top level domains of any

website. So, that means if we need to

find out where telesco.com is, we first

need to know which server is handling

all the .coms. So, in order to do that,

our root server will give the response

that okay, you can go to one of the TLDs

which is handling the top level domain

for .com and the root server will return

the IP address of that. Maybe this is

the IP address and from now, we know

that we are going to hit any TLD,

top-level domain for a .com. Now, from

this TLD, we are again not going to get

the complete IP address of the site. We

are going to get the proper server or

the name server or the authoritative

servers which are managing the all the

IP addresses for our domains. Let's say

this is the IP address of our

authoritative name server. And from

this, we are going to make another call

which are our authoritative name

servers. And from here, we are going to

actually get where our telesco.com

resides, the IP address, which was

19283841.

And once this is captured through our

browser, the IP, then it is going to

make the call to telesco.com server and

it is going to get the web page which is

desired for. I get it this can be a

little confusing at start, so let me

reiterate it for once.

So, whenever we click on any site,

whenever we want to find any site, let

it be telesco.com, it is first going to

our service provider, internet service

provider. Also, you can see uh

it can go through router and router can

also have DNS resolver. So, from

internet service provider which have DNS

resolver, this component is going to

resolve all our queries for us and

ultimately will give us a perfectly fine

working IP address for telesco.com or

any other site for that matter. So, the

first step is it will go to the root

servers. What is root servers? There are

13 companies who own these root servers

who doesn't know the IP address of your

site directly, but knows which TLD to

follow. So, from root server, it is

going to check what is your

TLD, top-level domain. And in this case,

it is .com. Now, as it is dot com it is

going to get the IP address of the TLD

which is handling the dot com domain. So

after this first call now we are going

to call the TLD which we get from the

root server and from this also we are

not going to get the complete IP address

of telesco. We are going to get on which

name server we are going to hit to get

the complete IP address of telesco.com.

So from here we are going to get the

authoritative name servers IP and

accordingly we are going to hit our

third and final call to our

authoritative name servers. Our

authoritative name servers are mostly

can be configured in GoDaddy or

Hostinger or any other service providers

in your current application. And from

these authoritative name servers we are

going to get the IP address of

telesco.com. Now in order to click one

side we have to do these three

operations. This can be heavy, right? So

this is only done first time and the

next time when you click on anything it

basically takes the value from cache.

Now cache can be stored in three

locations here or more than three if you

think about it, but three are definite

locations. The first is the DNS resolver

can have the cache to have all the

common IP addresses which we are going

to take or the all the common domains

which we are trying to hit. It can store

it here. Also on our system we have two

things. One is OS and one is browser. In

both of these areas we can basically add

that domain and the IP address in order

to get the details fast. So browser also

maintain that cache OS level also we can

basically maintain the cache and on our

DNS resolver site also a cache is

maintained in order to get the details

fast. Now when we are at this topic let

me tell you about one more thing, that

whenever we create domain name systems

like this dogs or subdomains, these all

entirely creates a zone

in the authoritative name server. That

means telesco.com including all the

subdomains. So, one zone can contain

multiple domain list, which will include

only the domains specifically to that

particular main domain. Okay? And

through this it is easier to get to the

main domain. So, in this case, whenever

we are hitting the root server, we are

getting the IP of TLD, again hitting the

TLD, getting the IP of authoritative

name server. And in authoritative name

servers, we can have different zones

attached to different domains.

And in this zone, if it is telesco zone,

in that case, all the domains which are

associated with telesco.com will be

here. So, if instead of telesco.com, if

we are hitting courses.telesco.com,

and from this, the authoritative name

server will get that this is our telesco

zone. And in telesco zone, we will get

the courses.telesco.com

IP address also. So, yeah, this is it

for DNS for now. I hope you understand

it. If you get it, let me know in the

comments. If you have any queries also,

let me know in the comments. Have you

ever wondered, if you see Zomato, if you

see Uber, if you see Ola, if you see

Rapido, and all these applications, also

Swiggy, they all have an inbuilt map.

But, did they create their own map in

their applications, or do they use the

existing Google Maps in order to do

that? Now, that is possible only through

APIs. So, Google Maps present some APIs

to other applications which they can

use. Now, in order to understand APIs in

detail, let's dig into this video and go

to our screen. If we think about API,

the API basically stands for application

programming interface. Now, what does

this mean? Suppose this is our

application.

Any application will work. Let's say

this is a movie application.

In which we have all the movies, plus we

have all the ratings, all the reviews of

the movie. Like IMDb or something, okay?

Like Rotten Tomatoes or something. So,

we are already capturing all the details

which includes a movie like latest

movie, latest hits of 2000, 2010, 2020,

everything. Now, we already have this

amount of data which is stored in our

database and our application is already

working on that. Let's say now in any

other corner of the world, someone is

trying to create one application based

on the data which is already captured in

this app. Now, they have two options.

Either they can create their own engine

like this, their own application where

they will again curate all the ratings,

curate all the reviews, and curate all

the data. But now, where are they going

to curate this from?

For this, every application leaves some

end points which can be used by another

application. Now, if

for our application or for this

developer application, if they want to

use rating of some movie application,

they can directly call the APIs of this

application and through which they will

get the proper information which is

required for them. Now, like this, this

developer or this application don't have

to create their own application for the

task which has already been done.

So, they can use the application data

directly. But again, can't we directly

give access to our database? This can be

dangerous, right? Because then this

application again have to handle a lot

of scenarios based on the database

queries, plus they can also delete some

important data in this database which

will affect the other applications which

are using the same content. Also, there

are some added functionalities. Let's

say we are giving an option to write a

review again. So, in this we have

already handled that case through some

or the other API. So, instead of

creating a new endpoint in this

application, we can directly access the

endpoint of the other application, which

is going to take our review also.

So, in this case, this application saves

a lot of time because they are using

another application, the APIs of another

application through which they are going

to do the same work. Now, in this case,

if this application wants to add some

data into their database also, they can

easily do this. So, this is one use case

of creating an API or using APIs of

other application. Also, there can be a

case where you're creating an

application and in the backend you are

exposing the APIs which can be used by

your front end. So, front end can be a

web application or even a mobile

application. Can be both also. Let's say

we are creating some backend and from

for that application we are also

creating front end maybe using React or

something and backend using Swift or

something. In both of these cases, these

languages, the React and Swift, both are

capable of handling APIs. So, they don't

care about which language did the

backend use in order to provide them the

API. They will directly call the APIs

and use the information which they are

trying to use. So, this is another

scenario where there are no two

applications, there is only single

application which is divided into front

end and backend part and using the data

which backend is providing. Again, in

this case also, you can think that they

will use the database directly, but it

is restricted and prohibited for the

security of data which we have already

discussed in the first case. So, with

this we understand the core

functionality or usage of the APIs.

Number one is if you want to give access

to some other application, and number

two is if you want to give access of the

endpoints or the APIs to our front end.

Now, this both can also happen at the

same time because it is up to us how do

we want to expose our APIs. Some

applications or some backend basically

restrict the APIs to be used through

their front end applications only. And

if you want to make them open, you can

make them open so that other application

can also use the same API. Now, let's

move forward to the type of APIs. The

first type of API is called as REST API.

If you are a developer, you already are

using this. Also, if you are creating

projects, you might already have created

them or use them in your application.

Now, REST basically stands for

representational state transfer. REST is

one of those APIs which you will see in

most of the application nowadays because

of its easiness, its structure, and its

ability to easily maintain a large

amount of data with low volume.

So, that's why we use REST. Mostly, in

REST we use JSON,

which is JavaScript Object Notation,

which is a way or a format through which

we will send or retrieve the data. It is

a easy to use format, that's why we are

sticking with this for this today's

video, and we are going to cover this in

details in a minute. Also, we have some

other types, that is SOAP. SOAP

basically stands for Simple Object

Access Protocol. This is also one way of

handling the data across applications,

across different platforms, and SOAP

basically use XML formats in order to

transfer the data.

XMLs are replaced by JSON because XMLs

were really bulky in nature because in

order to create every object or every

array, we have to mention the attributes

again and again, which is solved by

JSON. So, that's why we prefer JSON over

XML any day in today's time. But, the

SOAP or the XML was famous back then.

So, that means there are a lot of

applications which are still using XMLs.

Why? Because there are legacy

applications. They are built way earlier

when JSON was introduced and still are

using XML. So, in legacy system, if the

APIs are created and exposed using XML,

that means

in order to make our system work, we

also need to know about XMLs so that we

can integrate those APIs into our

system. Also, you can use some

middleware application where the data

will be taken in the form of XML and

will be converted into JSON and your

application can use that data

afterwards. So, that's why if you are

working with a legacy system, it is

preferred to understand the XMLs also so

that you can convert them in JSON or you

can work them without converting them as

well. So, after SOAP, let's jump to the

third one, which is GraphQL. So, GraphQL

is also used widely across many

applications, many platforms. So, the

basic thing in GraphQL is GraphQL

basically says that I'll provide a

single endpoint, only a single endpoint,

and based on that, you can send

whichever query you want. So, in FE,

from FE side, we will not hit multiple

endpoints, but we will send a request as

a query. That's why the GraphQL stands

for graph query language. So, GraphQL

also have a query language that is

similar to SQL. In SQL also, we have

multiple queries from which we can do

our things.

In this also, only one point is exposed

as I told earlier and using that only

the front end now can send multiple

queries and access that particular

endpoint and get the data which they

want. So, this is the core concept of

GraphQL. So, as I said, GraphQL is also

present in our today's time also. So,

there can be a chance that you learning

GraphQL and working on it. The other

type of API is gRPC. Here, the RPC

stands for remote procedural calls and

for G some may say that G stands for

Google because it is created by Google,

but it is gRCP

remote procedural calls only. So,

instead of using XML and JSON, we have a

new format here which is called protocol

buffers. It is very much efficient than

JSON and XML combined. So, that means

the size of protocol buffers are usually

very small than XML or JSON. So, that

will enable the data to transfer faster.

So, because of this only Google created

this and uses this.

So, this is small in size and faster to

transfer. In today's applications where

we have microservices, right? So, in

microservices, this is our entire

application and let's say our

applications is divided into multiple

applications which is combining the data

of their endpoints through each other.

Let's say these are the connections and

through this they are going to provide

the actual API to the outer world. Now,

for this communication between these

microservices, we often use gRPC. Why?

Because internally at least the

communication should be faster to

provide the relevant information in a

low latency. This will also help us

having multiple microservices and having

various kind of communications between

them before giving the actual data to

the other application or the front end.

Now, let's jump to the last one in our

list today, which is WebSockets.

WebSockets are also used in today's time

and today's application for various

purposes. For every notifications, if we

see, WebSockets are used. In order to

maintain a chatting stream, WebSockets

are used. Wherever you need some

real-time connections, WebSockets are

used. So, its architecture is slightly

different from the other services. In

other services, front end or the other

application will place some request to

the application and then they will

provide the response. But, in

WebSockets, whenever front end creates a

request, a channel is created. Now, once

the channel is established, then the

conversation can be lead through back

end also. Whereas, in other forms, only

front end or the other application,

which is trying to access the

information, they send the request and

get the request. But, in this case,

since the back end also can initiate the

conversation, that means it can also

send some or the other information by

itself. Those information can be the

chat or the notifications, which is very

helpful for front end to get. Because,

if this channel is not there, the front

end have to constantly ask the back end

whether there are some new notifications

or not or whether the chat is there or

not. But, once the channel is maintained

through WebSockets, it is easy to

navigate message from back end to front

end and front end to back end as well.

With this also, if you have played

quizzes on our Teleschool platform only,

Teleschool channel, in which you may see

an application which Naveen sir presents

in order to share the quiz with you all,

you use that application in order to

give the responses. Now, those responses

or the marks are calculated based on the

response time and the correctness of the

answer. Now, those connections are also

maintained through WebSockets. That's

why you see many notifications or some

GIF or something something. So, this is

about web socket and with this I would

like to conclude this session because it

is going to be a long session and it is

better to understand these things in

parts. So, these are the most used five

type of APIs which we have. Now, in

order to understand them briefly in the

next video, I'm going to talk about

RESTful APIs in which we will discuss

the various aspects or the parts of the

APIs

and also see what are the

recommendations and usage of those APIs.

Now that we already know about RESTful

APIs and all the other different type of

APIs, it's time to deep dive into one of

them. And for today's video, I have

selected REST APIs to deep dive into.

Why? Because it is the most used APIs

across the world. It doesn't matter if

you are starting a career or you are

already seasoned, you have to work with

RESTful APIs for sure. So, let's move to

the screen now. RESTful APIs are also

known as REST APIs. As we know, it

stands for Representational State

Transfer. What does it mean? It means

that REST is also one format through

which we will transfer our data from one

end to another end. What are these ends?

We have discussed this already. Either

we are giving some data from front end

to back end or vice versa. It is also

possible that one application needs to

get some data from some other

application. So, in that case also, APIs

are used and REST specifically is used

to transfer some data from one end to

another end. So, basically, it is a way

of how our applications are going to

communicate with each other. Now, before

starting, let me tell you that in REST,

we normally use JSON. JSON is a standard

way of sharing the data.

In JSON, we can have array, we can have

object and with these two combinations,

we can share our data in the form of key

and value. So, let me create a simple

JSON. Let's say this is a start of an

object and here we have a key, let's say

name. For value, I'm writing my name.

With comma, we separate different

values. So, after comma, I have to write

some other key. So, let it be age. And

with age, I can write any numeric value

as well. So, as we can see, this is

string value and this is a numeric

value. There are more values who are

accepted as a form of value, which is We

have already seen string and then

number. We can also use boolean, even a

nested object, arrays, and a null. Null

can also be used because if we don't

have any value for that particular field

directly, it is easy to say it is a null

value. And the application which is

using that value can take care of it.

So, this is all about the JSON which we

need to know. We need to know that it is

in the form of key-value pair and what

are the values which we can use. Now,

let's move forward to the next part,

which is endpoint. In order to

understand endpoint, we have to get two

things. One is method and the other

thing is path. So, in order to access

any data, we have to have some URL on

which we will hit with that particular

method and based on the method selection

and the path, we are going to either get

or pass some data to our server. Let me

start this with an example. Suppose we

are using HTTP as uh any site, my site

.com. So, this is our main site and

after that, we can put a slash and use a

different URL. Let's say we are

separating all the APIs in the API

subdomain. So, APIs is there and then we

can also segregate the APIs using

version. So, this is version one and

here we are going to write resource.

Now, resource can be anything. If I talk

about any e-commerce application, it can

be a product, it can be coupon, can be

user, can be anything, basically. A

resource is an entity through which we

are navigating and getting and passing

the data to the database. So, this is

the URL. The URL is called the path

through which we are going to either get

or pass some data. And with this, we

need to have a method. So, for method,

we can have a get basically. So, get

method with this path combinedly creates

an endpoint and through which now we can

get some data. We expect this path to

retrieve some data because we are using

get. Now, there can be multiple ways of

creating this URL. So, normally, we hide

the domain part of it. We don't use them

in the explanations. We only use the

resource part. Also, we can remove the

version also. With that, API can also be

removed in order to understand the

resource part directly.

So, from now on, if I create any path,

let's say users, it is expected to have

all this domain name API versioning if

it is there beforehand. So, I'm not

writing that again and again. Cool. So,

with that, if I write users here and use

get method, now I'm expecting some

response from our server. I'm expecting

either a list or a response from our

server. We'll come back to responses in

a while, but this is how we can get some

user information from our server. Now,

let's move to the second part. In this,

we don't want all the users, we want a

specific users. So, to find a specific

user, we have to provide them an ID.

Normally, we provide them in the URL

itself. So, it is get users and here we

can include the ID. So, in real time, it

can be used as users/1.

So, if we write one, the user with the

ID one will be retrieved. Now, with

this, we can also add any user. So, the

third part is

adding a user. So, again, the URL will

be same users and now we are adding the

user, so we are using the method post

here. So, in this with post, we are

going to give some body, and in body, we

will have some JSON which will store the

user information which we are trying to

create. And based on our request, our

user will be created in our database.

Now, once we have all the user details

in our table, we may choose to update

any user information, correct? So, for

that, we use another method which is

put. Again, the URL will be same, users.

And with this users, we will send the ID

of the user of which information we are

trying to update. So, with put, we also

have another method which is patch,

which can also be used in a similar way

to update the user information. Now,

what is the difference between put and

patch? It is simply that if we use put,

our entire user object will be replaced

with the new data. So, this is going to

replace the user data. And if we don't

want to update all the fields, we want

to update a specific field or particular

field, suppose username, name, age, or

anything, so we can use patch. It can be

used to partially update the user

record. Let's understand this with an

example. Let's say we have any user

information which have ID and have a

name and have a username, maybe age.

This is all. So, these are all the

values which are

user currently have. Let's we put some

values. So, if our user table or user

entity have these all four values, if we

use put to update a particular change

only, let's say I'm going to change my

username from Akshay to AK something.

So, now this change will only be done if

we again send the entire body into the

put request. If we only send username

like this, then all the other values of

our entity will be updated with the

default value. So, if we send this kind

of data in a put request, it is going to

replace our entire entity. So, that

means the name, the age will be replaced

by the default or null value. So, it is

not recommended to use put entirely if

you want to update your entity

partially. In those cases, always try to

use patch. In patch, we're going to

again update the particular information.

So, this is perfectly valid for a patch

request. So, with this, let's move to

the next part. The next kind of method

which we use is delete. In delete also,

normally we pass the ID like this. So,

if we write users and with {slash} we

write one, so that means the user with

the ID one will be deleted. Now, this is

the usual stuff. You will get this

information anywhere. Now, let me talk

about something which is nested data.

Let's understand nested data with an

example. Suppose you're creating a blog

site. So, in blog site, we normally have

our users who are going to read our

blogs. Then, we have blog, the actual

content on the site. And on blog, the

user can comment. So, we have clearly

three entities: users, blogs, and then

comments. So, these three entities are

interrelated. So, a user is going to

give comment on a particular blog. Users

can't comment if there is no blog,

correct? So, this is the way. Now, in

order to create APIs for this, we can

create a users API for all the users'

activity. So, all the users' activity

means we can use it for get, put, push,

delete, and patch also. Now, this is

simple because a single entity is there,

users, and all the operations are on

that users only. Now, let's say we want

to create an API on which user can add a

comment. Okay? So, for that, what we can

do, we can either go from the blogs

perspective. So, every blog have a

particular comment section, right? So,

blogs, then here we can write ID. So,

this ID is basically belong to the blog,

and with blog, we can write comments.

And if we write get here, what it is

going to do, it is going to give us all

the comments which are associated with

the blogs with ID this. Correct? So,

this is a nested relationship where

we're trying to access comments, but we

are going via blogs. We can also do like

this. Suppose we are on the user's page,

and we want to see all the comments that

user have created. So, we can again

create a get query

via users, where this is the user ID

here, and we can see comments. So, we

are dealing with two different entities

by creating the relationships between

them. So, by this, we are going to get

all the comments which are created by

the user with this ID. So, we can go to

comments via blogs, as well, and via

users, as well. So, this is nested.

Again, if you want to update any

comment, delete any comment, we can use

simply the comments path

where the user can enter the ID here of

the comment.

And they can use it to update or delete

that particular comment. So, here you

can also ask me that why are we using

this kind of nesting in URL? Why are we

not sending the data in the request? So,

that is also possible. We can do like

this. We can

go to this comments, and we can pass the

request body, in which we'll pass

the post ID to get all the comments

associated with this post. This is also

possible, but this is not recommended.

As there is a direct relationship

between two entities, we don't normally

go for this body approach. We go for the

URL approach because it is easy to

understand also. So, when do we go for

the body approach? So, the body approach

is used when we are trying to filter any

data. So, we understood two concepts

here. First is the nesting data versus

filtering. So, when to use which one? If

you have a clear relationship between

two entities and it is clean, so you can

go for nesting. And also in filtering,

apart from sending the data like this in

post ID, you can also use

the URLs like this, comments, and then

in query parameter, you use post ID.

Now, this is again not recommended

because there is a direct relationship

between comments and post. So, it should

be a nested type of URL. So, if there

are clear relationship, we can use

nesting. And if the relationship is

complex, we can go for filtering. Also,

the filtering will be used whenever we

are trying to do pagination or we are

trying to filter particular records

based on multiple ways like we can go

for any e-commerce page where we are

seeing different products and we can

filter our product based on our color,

price range, and all these things. So,

those things normally go for query

parameters as a URL, right? And we don't

go for nesting there because it will be

complex to handle a long path with

nesting data. So, the crux is if the

relationship is pretty straightforward

and we will go with respect to the other

entity, we'll go for nesting kind of

URLs. And if there are complex

relationships or the filtering is used,

we'll go for the query parameters. Now,

let's move to the next part. How do we

pass the actual data to the endpoint?

So, there are three ways. Either we can

use path, so we can integrate the IDs in

the path, or we can use some query

parameter, or we can use body. I have

given you glimpse of these all three in

the previous section, but now let's talk

about this specifically. So for path, it

is very specific that we will only pass

the recommended IDs or maybe slugs or

maybe that particular unique identifier

only. So this is unique identifier. It

can be ID or slug. Slug is basically a

short form of that particular post. If

you are using any blog site, so let's

say the blog site the blog is about what

is Java and in slug you can see

something like blog /what is Java like

this. So this is our slug. Again, this

slug will be unique throughout our

blogs. So the name of the URL I have

also written blogs, it is wrong.

Normally we address the URL in the form

of plurals. So it is going to access in

multiple areas, multiple scenarios. So

normally we don't go for blog, we go for

blogs like this. Cool. Now for query

parameters, whenever we are filtering or

maybe sorting, we can go for query

parameter. Suppose we are again

scrolling through our blogs and in blogs

we can write like sort ASC or we can

assign some numbers also. So ascending

means sort ascending maybe by date. So

whichever date they are created, they

will be sorted by that. Or maybe we can

use DESC for decreasing order. So that

the latest post will be pop up first.

With this again we can use the query

parameter for filtering. So maybe we can

use blogs again here and with that we'll

pass query parameter. So this is Q for

maybe query search or search step. So

anyone with is searching for Java and it

will give us a list of blogs for Java or

which have Java in their name or their

content or their category. So, this is

all about query parameters. Now,

whenever we pass any information in

query parameter and path, those

informations are going to be exposed,

right? So, that means we can't pass any

sensitive information into query

parameters or path. So, for that we have

body. So, let's say we have a login

path, and now we can pass username and

password in the body, which is not

exposed at all.

So, it is safe. So, the rules are pretty

clear when we use path, query params,

and body. Now, with this let's see a

full request in action. The first thing

is we will have our method. So, this is

maybe a post method, and we are using

some API of version three, maybe.

For version three, we are using users,

and then we can also pass the metadata.

Suppose content type we can pass, which

is application/json

here, because we are passing the

information in the form of JSON. We can

also use XML, or if you have latest

information, we also have a new format

in town, which is

tone. So, we can also use various kind

of information, but because currently we

are using JSON, so we can pass it like

this, and with this we have our body. In

body, we can pass anything, maybe some

data we are trying to add, and then

maybe username. So, like this we can

pass the entire request or the data to

our server. So, this is what? This is

our method. This is the path. This again

is metadata, and this is our body. Now,

once we have understood the request

correctly, it is time to understand the

responses. So, in order to understand

the response, first we need to

understand the response code. So, there

are specific codes assigned to a

specific type of response, which if the

user gives will clearly understand what

they want to tell us. So, let me start

with those status codes and then we will

move forward. So, these are all the

response codes which we have to know.

200 stands for okay. So, this means

everything is working as expected. Don't

worry about it. Then we have 201. 201 is

basically used when there is a new

entity created in the database. So, this

is created. Then we have 204. 204 is

again saying that everything is working

fine, but I don't have anything to

provide to you. Let's say we are

deleting any entity, so it is

successfully deleted, but they don't

have any response body to give to us.

So, that means there is no content to

give to us, but the request is working

fine. Then we have 301. 301 is basically

used for redirects and 301 indicates

that it is a permanent redirect. And if

the redirect is not permanent, then we

use 302. That is temporary redirect.

Then we will come to the 400 series. So,

all the 400 series basically says that

there is some issue with the request we

are having. So, the 400 stands for bad

request. That means some or the other

data is not correctly mentioned in the

body. Either they have some validation

issue, either either they are missing

when they are required. Either they have

some other issues, but request which we

are getting, the body doesn't have the

correct information. So, in that case,

we normally give 400 bad request as a

response. Then we have 401. 401 is given

when someone is trying to access some

information for which they are not

basically authorized. So, 401 is

unauthorized. With that we have 403,

which is used for forbidden. Suppose on

an LMS, you have not enrolled for a

course and you want to access a

particular video or lesson part of that

course which is not publicly accessible.

So, in that case, you will receive 403

forbidden because you are not allowed to

see that particular resource. With this,

we have 404 page not found. This is

basically used whenever the URL is

incorrect or trying to access some or

the other resource which is not there

into our system. And at last, we have

500. So, 500 is the response which is

given to a front end when there is

something wrong with the back end. So,

this is related to back end. Some or the

other error is there in the back end.

Maybe some syntax error, maybe some

other error, some logic is missing, some

data is missing, something is wrong with

back end. So, this is internal server

error. There are many errors which we

don't want users to see. So, those all

are handled by 500 because we don't want

the user to understand what is the issue

we are facing with our database

connection or anything. So, we will

create logs into our system and through

which we'll resolve the 500 error

issues. So, these are all the most

commonly used response codes which we

have to understand. So, now whenever we

are creating any APIs, if the response

is completely fine, we will provide 200.

If we are creating any entity, we will

provide 201. If we are deleting or

performing some operations where we

don't have to give any content or any

data or any body in the response, we

will use 204. Then, if the permanent

redirection is there, we will use 301.

If the redirection is for temporary

basis, we will use 302. For any bad

request in the body, we will use 400.

For authorization, we will use 401.

For forbidden or the content which

requires some special access, we will

give 403. 404 we will give when they are

trying to access any resource which is

not there and 500 we will give when

there is any issue in the internal

server. So with this, let's move to the

next part which is response body. So

normally in response body, what is

allowed first we start with that. So we

can return an array, right? Also, we can

return an object. With this also we can

return data as nested objects. Now in

most of the applications whenever you

are going through any YouTube tutorial

or any course also sometimes when you

are dealing with multiple responses,

let's say the responses for this is

normally an array, right? But it is not

recommended to handle the response as

array. It should always be wrapped in

the form of an object. And if you want,

you can use any other key for this users

and in the value you will give these

responses. Now these can also be string

or an object. Normally an object only of

all the users, but it should be handled

like this. So this is the correct or I

would say recommended way. So every

response body should be an object. And

even if we're trying to give an array,

we have to wrap it up into an object and

give us in the key value form. So this

is a good practice and through this

we'll achieve a more long-lasting code.

Why? Let's say today we are passing the

user details only and tomorrow we

understand that okay, we have to add

more fields. So we don't have to change

much on the front end side or the

receiving side, but we will simply

update our response, change the object

here

and maybe add some other field user

count and pass that information maybe

29. And with this we can end our object

here. So we can pass as many

informations as we want, but if we have

used a single array

as a response, then we have to struggle

with it and we have to change the entire

architecture of our response. So, it is

always recommended to pass the response

always in the form of an object. So, let

me write returning as array not

recommended. Let me know in the comments

if you already knew about this or you

are hearing this for the first time. So,

with this, let me create a map of

request type and responses. So, let's

say here

we have request. Here are the responses

which we are going to receive. So, if

the request type is get and we are

trying to access many records, so maybe

we are trying to access users, it is

going to give an array of records,

right? So, it is normally going to give

200 as a response code and going to give

an array. Now, we understand that we

have to not give array directly, so it

will be wrapped in the form of an

object. Now, the next request is also a

get request where we want to access a

particular object. Let's say one. Now,

you can think that we will give the user

information directly, but again, it is

not recommended to give that directly.