2:33
2025-01-06 13:08:28
3:21
2025-01-08 09:13:02
4:34
2025-01-08 09:31:44
22:26
2025-01-08 09:34:32
22:27
2025-01-08 10:16:32
24:59
2025-01-08 11:36:15
6:37
2025-01-08 14:09:18
17:55
2025-01-08 18:20:43
16:28
2025-01-08 18:20:43
34:39
2025-01-08 21:21:43
34:51
2025-01-08 21:21:44
Visit the Kafka for Administrator course recordings page
United Arab Emirates - Kafka for Administrators
WEBVTT
-->
So memq has been built on top of Kafka by the Pinterest.
-->
So they were trying to do the large file management using Kafka.
-->
So basically like which is a better management pops up platform compared to Kafka which
-->
basically like saying that it can handle more GB of traffic and with better file management.
-->
So if you see this memq, memq internally uses Apache Kafka.
-->
Any questions or any clarifications still?
-->
Any doubts till now?
-->
One is like one of the architecture of the Pinterest.
-->
So they were actually using large file management, so which they usually get it to Kafka and
-->
then send it to S3 and kind of manage it.
-->
This one is not solving their problem.
-->
What they did is like they developed the memq, memq is on top of Kafka which handles
-->
it better in terms of storage and all those.
-->
So this is developed by Pinterest because they were having issues in terms of managing
-->
the large files using the pops up model.
-->
What they did is on top of this Kafka framework they added the storage.
-->
You can add Amazon S3 or GCP or anything.
-->
Now this is managed by itself in the memq.
-->
So you can go through this whenever you have some time to understand, if you want to
-->
better understand how the other things will look.
-->
Similar to what we have Ukip or they have NetE which handles all this stuff.
-->
There are n number of things that came on top of Kafka, but Kafka on a core.
-->
Any other questions?
-->
No.
-->
And with that, I think most of the topics are discussed.
-->
I just sent you the chart like where the whole set of topics that we are discussing.
-->
You can go through them.
-->
You can see any one of them you want to go again, I can go through it.
-->
Most of the things we have covered.
-->
What is the traffic usually you have any idea like in terms of input traffic?
-->
You see here in 2020 they have a peak of 25 GB inbound and 50 GB of outbound from
-->
For this they have almost 50 clusters.
-->
It's very similar to e-commerce.
-->
But I think this is more of a social media right?
-->
They got more data that is not structured.
-->
All these small forms and payments.
-->
All the files.
-->
Even if you put it in a topic, you need to scan right?
-->
You need to make sure it's not a virus.
-->
They have one point which says they usually use their drives.
-->
Then they switch it to SSDs for better increase of their operations.
-->
They talk about the rebalancing, how they do the rebalancing, how it improves.
-->
And also the message format conversions like one format, other format.
-->
They are actually on AWS.
-->
So it is a good read to understand the cause, the data associated with that.
-->
And they talk about the ISRs, how they replicate.
-->
Most of the things we have discussed are there.
-->
But you can understand how that specific use case is having problems on how they
-->
solve it.
-->
And how they created a new main queue.
-->
I'll just send you the name.
-->
There's no exam vouchers for this right?
-->
There's no exam vouchers right?
-->
Which one?
-->
Exams vouchers.
-->
No right?
-->
No.
-->
I need to ask.
-->
Is there any certification for this?
-->
Do you have any certification for this class?
-->
We don't have right?
-->
Do we have any certification kind of thing that says that they...
-->
I will show you the e-certificate after the thing.
-->
Here you do some survey right?
-->
Yeah.
-->
Send it, send it.
-->
That's all we want to know.