2:33
2025-01-06 13:08:28
3:21
2025-01-08 09:13:02
4:34
2025-01-08 09:31:44
22:26
2025-01-08 09:34:32
22:27
2025-01-08 10:16:32
24:59
2025-01-08 11:36:15
6:37
2025-01-08 14:09:18
17:55
2025-01-08 18:20:43
16:28
2025-01-08 18:20:43
34:39
2025-01-08 21:21:43
34:51
2025-01-08 21:21:44
Visit the Kafka for Administrator course recordings page
United Arab Emirates - Kafka for Administrators
WEBVTT
-->
I just want to do one more thing you see it here the query or the metrics browser.
-->
So usually they use count they do all the stuff if you want to explore more or more
-->
matrices on how to handle it we need to learn about how we can handle a graph on a but
-->
I am using a sample example of one of the metric what is a partition count of
-->
how many partings are there on this server okay since this is saying like in this
-->
clusters I have only one broker that's where it is going one okay one server
-->
yeah if you see previously there was no request now since they were you
-->
want to see the data in your thing okay Janice is able to see it yeah
-->
once we do some action on the topics all this stuff then you can see the data coming in the
-->
click on it edit it and you will be able to try it out and let me know if we have issues
-->
everybody able to see the data coming in somebody does any operation there like kind
-->
of getting the messages in and out then the data will appear and there is a delay
-->
1 to 2 minutes but monitoring is primarily
-->
So, any questions any doubts or any issue while you are doing it?
-->
No, this is actually coming from the confluence, yeah, that's my cloud, this one here, which one, this one, this one is in your local, but data is driven from the cloud.
-->
Right. Yeah, you're sitting there, you can search a dashboard, my dashboard. Oh, if you want to access this dashboard, it should be in the cloud, because this is my local.
-->
If you want to do it kind of like somebody you want to show right or if somebody has to access, then you need to put it in your cloud.
-->
Then tie it up, then people can say from there you can, but this is my local you cannot be able to, you will not be able to access it, but.
-->
So.
-->
So.
-->
Yeah.
-->
These are open source, the only thing is you need to have the hardware, ideally a VM to run, you can run both of them on the same server.
-->
But in general practice right, they run Prometheus on one server and Grafana on another server to make sure if something goes wrong, at least you have the data on the Grafana.
-->
Yeah.
-->
This is in general like you can use, there are so many tools to visualize but Grafana is the most popular one for visualizing because it's an open source it has.
-->
You can see like people building all the dashboards we can utilize them or you can take an inspiration from them and build our own build our own dashboards for monitoring it and trying to see if something is going wrong.
-->
Yeah.
-->
So in my previous order we used to have the, how many orders are coming in, how many orders are going down.
-->
So on a high level.
-->
We used to use that.
-->
I mean there's like no direct simple way.
-->
Yes.
-->
From the.
-->
Yeah.
-->
There will be no simple thing but yeah, like, if you want to get something you have to use something like this tools.
-->
Are you utilize the conference load itself, but they will have limited number of things they will not have the more like to see this one.
-->
Here we have a limited list throughput production the major ones usually what they do, but not every matrix will be there. So it is very limited.
-->
Well,
-->
the audit loss. No,
-->
you will not be able to get audit logs from the
-->
conference because they will not allow you to get it
-->
until unless you have your own cloud. I mean, this one you installed in your.
-->
There are two ways to do it one.
-->
Get the cloud from the conflict or use the conflict cost cut to install in your own on print. If you do it on time then you'll able to do it because you are holding the loss.
-->
Yeah.
-->
If you have those logs.
-->
No.
-->
Yeah, if you are. That's what I'm saying. So yeah, that will do actually that is another session we can do it.
-->
But conflict, I'm not sure how they do it or how they pull it.
-->
But if we have our own Kafka broker in our on frame or in our cloud.
-->
By default Kafka emits some of the matrices, I will show it on the next session.
-->
So basically you can install it to see the.
-->
Have, have added for the next session.
-->
It's called
-->
JMX basically Java emits some so Kafka is built on Java.
-->
Since it is on Java, what we can do is we can use this JMX Prometheus Java agent to publish to one of the thing.
-->
And from there, we can put those details to the Prometheus, and we can manage whatever yourself right.
-->
How many people, I mean, how many topics got created, whether we have modified something, but that we need to use.
-->
You need to have to write the rules.
-->
Seeing that what metric you want to capture and from there, what you want to take it up.
-->
There is no direct way as of now.
-->
So we have to do kind of like this.
-->
Let me show you.
-->
Yeah.
-->
Yeah, Melon maker is helpful to replicate or.
-->
Let's say I have one cluster of.
-->
Servers in one location.
-->
And I want to replicate it, let's say in an on frame.
-->
I want to take it the whole.
-->
We can use a mirror maker to replicate the whole thing to the other cluster.
-->
Like the whole you can copy the whole content of the another cluster, another server or another cluster.
-->
No.
-->
So replication is done as part of in.
-->
It's right now.
-->
Okay.
-->
Even we do mirror.
-->
You do mirror.
-->
Mirror in the sense.
-->
Inside a cluster.
-->
Let's say I have five.
-->
Brokers knows how to replicate.
-->
If it is a Joe Piper, Joe Piper says if it is fast, they internally know how to replicate.
-->
But if this all to the different cluster, which is residing in.
-->
Maybe in different cloud or in different data center.
-->
Then you want to replicate from here to here.
-->
Then the bridge is the middle.
-->
So.
-->
The maker is useful to do between two places.
-->
All together in a different.
-->
I mean, somebody's.
-->
I mean, one is managing, let's say one is managing five servers.
-->
The other is managing a different region or a different data center or a different flow.
-->
But if you want to make up both of them same and use kind of.
-->
Replicator is helpful in.
-->
I mean, the replication, whatever happens in between the brokers, it is done by or usually taken care by Joe Piper or.
-->
But if you want to do it in a different location or something like that, this cannot manage.
-->
Yes.
-->
You talked about the how many people are the matrices in terms of audit loss, right?
-->
You can do that, but.
-->
You need to kind of write rules.
-->
What type of data you want to take it out.
-->
And how you want to do it.
-->
Then based on the rules.
-->
The GMX will emit those to some from with a server and from there you can put it into.
-->
So we need to kind of.
-->
Add similar to this.
-->
As of now, like.
-->
But audit logs to the Grafana, I haven't done anything.
-->
In my arms, but that we can kind of.
-->
Take it and.
-->
But in order to do that, you need to make sure that the audit logging is enabled and the audit logging is there.
-->
I'm not so confident is giving you the audit logs.
-->
But they will have their own satellite.
-->
They can actually go into the server and it is being administered by.
-->
They can like go to their own Kubernetes cluster, log into it and see what is happening.
-->
But I don't think they will give you the.
-->
I don't think we are asking for the control plane.
-->
I'm just asking for the portal activity logs.
-->
Oh, the admin activity logs.
-->
That should be.
-->
I think I only see.
-->
Yeah.
-->
Those logs I don't think.
-->
We have it.
-->
I mean, since from my side, I don't have it.
-->
But generally, if you still want to know, like, what is happening on this, that is basically on a user management.
-->
Right. So the.
-->
There are two different audit log site.
-->
One, what is happening on a cluster?
-->
The other thing is, who is actually doing.
-->
Yeah. Yeah. So which one you're checking.
-->
You want both.
-->
Maybe like we need to check them like how they do the audit logs, all this stuff we can ask them to get it.
-->
Generally.
-->
They have the.
-->
General logs like what is happening on top of the cluster.
-->
But if you want to know which user is doing what.
-->
I'm not.
-->
You can ask conflict to say, hey, can you give me the user audit law?
-->
Then probably they may probably.
-->
But general logs will be audit logic.
-->
What is happening for a topic?
-->
What is happening for a cluster?
-->
Whether modified all the stuff.
-->
But who is managing it and who is using it?
-->
Probably they can't forward, but.
-->
I don't know, like, which plan you're using based on the plan they may have or they may not have it.
-->
We need to respond.
-->
Yeah. Based on the plan, they may give you the.
-->
If you see if you want to really.
-->
I'm just using this for.
-->
But if they are using any standard dedicated premium kind of thing, then just definitely they should be having the logs and probably the report.
-->
So we can ask them to emit them to some our own.
-->
So with this or the final server, then we can take it kind of give it what is happening and put a lot of.
-->
Yes, I also want to show how we can do the other things here.
-->
So let's say in this dashboard.
-->
OK, if something is going, we want to specifically do some alerting mechanism, like send them a kind of thing.
-->
So we can do add other tools.
-->
Saying that what type of data source you select and we want what data or dashboard, then choose what is the point and search something like that and create a rule and add it.
-->
Let's say.
-->
Let's say I'm saying this is my data source and I want this metric to be calculated.
-->
Then click on that.
-->
Then you get the query and add it.
-->
And on top of it, you select the room and kind of add it.
-->
Then do what you want to do with them.
-->
OK, whether you want to do say between something else or kind of them.
-->
Then.
-->
The Fana has its own inbuilt SMTP server.
-->
You can actually send them a message to any one of the configured units.
-->
That also can be done on the phone.
-->
If you are using a phone, it is like a useful alerting mechanism.
-->
What are we are getting from the confluent dashboard, right?
-->
Those are very limited.
-->
But if you still want to go ahead and further dive deep into more matrices.
-->
This is where you are doing like the document, which I have added right monitoring.
-->
This emits a lot of matrices like broker matrices, which I have discussed yesterday.
-->
Some of them.
-->
And you can put alerts on top of it.
-->
These are all the brokers are a meter.
-->
There are a number of matrices, which is like you can check.
-->
I should.
-->
I was saying like how we do the batch size, the average batch size max.
-->
We're using this matrices when you do any performance test.
-->
By using this matrix, we can increase the batch size, decrease the batch size, and you can select how many partitions you need.
-->
What's the throughput in general?
-->
This is what if you're doing any performance testing on the stuff, we use these matrices to manage it.
-->
You see here we have all the matrices like you asked about this one.
-->
Probably we can ask them to see what is happening on.
-->
But this generally what is happening on a high level, but not sure in detail at user level.
-->
I am not sure.
-->
In the next session, we will see about the general issues.
-->
And see how we can mitigate them or how we can.
-->
Fine tune the cost per service.
-->
Is everybody able to get the.
-->
This is what everybody I am seeing.
-->
Are we good or you still have any issues.
-->
But, you know, you want to have that.
-->
Yeah.
-->
Are you using any kind of these dashboards or you're just using the default.
-->
Okay.
-->
They're just maybe like shading on a high level.
-->
I think there are some metrics like.
-->
Give me a number.
-->
You don't show it.
-->
So actually, there is an user management thing.
-->
Probably they can and user.
-->
But if they add user, there is a cost of society.
-->
So probably.
-->
Yeah, yeah.
-->
They should be generally doing it.
-->
They should be giving you access at least like read only dashboards.
-->
Maybe the same.
-->
Yeah, but.
-->
We want it to be on the same.
-->
I don't even remember.
-->
I'm sorry.
-->
Excuse me.
-->
Okay.
-->
Okay.
-->
Okay.
-->
Okay.
-->
Thank you.
-->
Thank you.
-->
Thank you.
-->
Thank you.
-->
Thank you.
-->
Thank you.
-->
Thank you.
-->
Thank you.
-->
Thank you.
-->
Thank you.