39:11
2024-05-06 08:57:50
3:11:28
2024-05-06 10:30:19
24:39
2024-05-07 07:20:20
19:49
2024-05-07 08:03:25
1:14:35
2024-05-07 08:35:13
15:41
2024-05-07 10:06:25
38:33
2024-05-07 10:37:03
2:48
2024-05-07 11:19:01
59:37
2024-05-07 11:33:56
6:10
2024-05-07 14:55:25
39:40
2024-05-07 15:02:44
9:45
2024-05-08 06:44:21
29:27
2024-05-08 08:09:24
2:51:40
2024-05-08 12:09:24
Visit the Apache Nifi - GROUP 1 course recordings page
WEBVTT
-->
Okay, perfect. So if you can stop all your processors, I just showed Brett a little trick
-->
on how to quickly stop those all. If you want to go right back to your main canvas, use
-->
your breadcrumb trail at the bottom. There you go. Right click on that processor
-->
group. Say stop. There you go. And right click again and say empty all cubes. Okay, perfect.
-->
Perfect. Alright, let's go back into that. So let's see. Everything is empty. Let's
-->
leave everything stopped and let's run it once. And then whatever the output of
-->
convert CSV to JSON, we're going to take a look at. Don't run the update file name yet.
-->
There you go. Hit refresh. Alright, success. Let's take a look at that update file name.
-->
Let's take a look at that queue right before list the queue. Go all the way to your left
-->
and let's look at the attributes. I can already see file name there. Okay. File name is
-->
inventory.csv. Perfect. Say okay. And exit out of that. And run the update file name
-->
one time and refresh. We're not going to put the file yet. We're just going to run
-->
it once. Yep. And hit refresh. Success. Let's look at that queue. List the queue.
-->
And why is there, okay, hit that. File name. Say okay. Exit out of that. And you're
-->
using an update attribute, right? Say configure. Oh, you capitalized file name. It's a lower
-->
case. So no, no, no. Cancel. Delete that. Delete that altogether. And then add a
-->
file name in the lower case because that's the property name. There you go. And now dollar
-->
curly bracket file name. Close your curly brackets. Awesome. Awesome. Awesome. And
-->
that should work. And so you want to clear your queue on the place.json file right
-->
before it. You want to clear that queue. And then run it once and make sure. Okay.
-->
So I have 12, 10, 10, 10. You all have time. Let's take a 15-minute break. Use
-->
the restroom and stuff. Let's try to get back here at 12.25 my time. 10.25 your
-->
time. If you get, you know, if you get a free minute, I'd like for you to go in and
-->
just kind of touch everything up. You know, clean it up. Those types of things.
-->
Let's take a look at. So when we get back, we're going to take a look at your
-->
data flow and go through any final questions. We still have more hands on to
-->
do. But now that we've got a couple of data flows in our canvas, we need to do
-->
conversion control. So let's take a break and I will see you all in 15.
-->
Give another minute for folks to return. But, you know, great job to everyone
-->
getting this flow working. There was a reason I chose this flow to go through
-->
this morning. It really kind of shows off some of the controller service
-->
capabilities. It's more than your standard Nafa flow of just picking a
-->
file up, unzipping it, and putting it somewhere else. We really started
-->
getting into some of those controller services, those types of things, as
-->
well as, you know, schemas, record readers, record writers. And you'll
-->
see a lot of this in, you know, a lot of your data flows will utilize these
-->
services. The beauty of that controller service was, you know, now we can
-->
reuse those components. We can reuse those controller services if need be.
-->
We can reuse that schema. So if we had, you know, CSV coming in from
-->
somewhere else, we can reuse a lot of this same data flow. So that makes
-->
it a lot easier for us. There's even more advanced capabilities under the
-->
hood when you start diving into this. The ability to have, you know,
-->
multiple Avro schemas, for instance, and, you know, those types of things.
-->
So I really like this hands-on experiment. What we'll do now that
-->
hopefully all of us are back, we'll kind of go through that flow, make
-->
sure everything looks good. We should have now two data flows on our
-->
canvas. And now, after we get through going through those, we will work on
-->
checking them into versioning control and what that looks like and how that
-->
gets configured. I know we have some DevOps and, you know, some CICD
-->
questions originally. So hopefully this will start flushing some of that out
-->
and we'll go from there. So that being said, let's quickly go through.
-->
Pedro, it looks like you've got yours working. Everything looks good.
-->
Awesome. And you've started naming your processors and labeling them.
-->
It's in its own processor group. You know, those types of things.
-->
So I like how you put error logout to the side. That, you know,
-->
I can quickly look at this data flow and I can distinguish the different parts
-->
of this. I can see where you're logging your errors. And, you know,
-->
your processors, a lot of your processors have human readable, human
-->
understandable meaning. So yours looks good. Any questions, Pedro,
-->
from yours? And if not, I'll take silence if not.
-->
Okay.
-->
Yes, go ahead.
-->
Because I noticed you had yours going diagonally out.
-->
I was trying to figure out how to do that. I couldn't drag them,
-->
which is what I thought.
-->
No worries.
-->
Let's see here. Let me bring mine back up. That's a good question.
-->
There is a lot of hidden tips and tricks on this.
-->
So if you want, you can go to the line and just double click.
-->
You see how I did that?
-->
So you can double click
-->
where you want it to bend.
-->
And you can bend your lines.
-->
So you can...
-->
You see what I'm doing is double clicking that line.
-->
Okay, perfect. No, good question.
-->
Go ahead.
-->
So because these VMs are so much like the C, I accidentally clicked one and dragged it
-->
and really messed up my canvas. Is there a Control Z in the canvas
-->
to undo the last thing you did? Or do you just have to try and click with the fallout?
-->
You're going to have to deal with a fallout.
-->
I do like that Control Z question.
-->
I'm going to take note of that one as well and see
-->
if it's possible we can do something like that.
-->
That's a great question. We are at the mercy of the latency of these VMs
-->
as well as proxies and everything else that you have to go through.
-->
But yeah, that's how you bend them.
-->
How do I get the line back?
-->
You want to do a
-->
line back.
-->
Let me see here.
-->
I'm trying to remember not double click.
-->
Oh, it was double click.
-->
Yeah, I'm about to say it.
-->
I got rid of one by just double clicking.
-->
And then
-->
once you get off of it,
-->
then you just double click.
-->
There you go. You just double click the spot where you put the bend.
-->
That was
-->
your old Sean.
-->
Yeah, unfortunately I was tied up with work until like 15 minutes ago.
-->
So I'm pretty far behind you guys now.
-->
Did you download that zip file?
-->
Okay, perfect.
-->
So if you can,
-->
you can go to that zip file
-->
or go back to your main canvas.
-->
And then the sample data flows, you can actually import that
-->
CSV JSON demo data flow
-->
and it will give you the complete flow
-->
just for FYI. So if you want to catch up, I would go back to your
-->
main FYI flow canvas, bring down a processor group.
-->
There you go, that one. And then say to the right, say upload. You know there's an up arrow.
-->
Click that and then go to that folder you downloaded.
-->
Right on your C drive.
-->
And click that one, say open, say add, and then double click to go in there.
-->
So I go through and build these to make sure
-->
for instances I get called away, I had work to do,
-->
I had a technical problem. You will still need to go through and configure your
-->
CSV to JSON, the record reader and record writer.
-->
But all the services are there, so you just need to go through usually and enable them.
-->
Yeah, yeah.
-->
It should be good for the rest of the day.
-->
No worries, no worries. And then when you can,
-->
you'll have to fix your CSV from directory because you'll need to know
-->
where to get that, so you'll need to update that, you'll need to update where you write the JSON,
-->
and also enable all your controller
-->
services. But you should be able to catch up pretty quickly in that flow,
-->
as well as kind of walk through it and see if you have any additional questions
-->
or anything else, you know, just feel free to interrupt me.
-->
Okay, yeah, I'll go through all these processors and try to configure them.
-->
Okay, yeah, and like I said, if you have any issues, you know, interrupt me and let me know.
-->
Alright.
-->
Any questions? I really like, you know, just a tip here,
-->
you know, you want to label processors that are, you know,
-->
like in kind. So, you know, you can,
-->
let's see, the pick up files and the set attributes,
-->
you may want to combine a label just to, you know,
-->
say this is where we're setting attributes or something, but I really like how you got all these
-->
labels labeled. I like the naming you put in.
-->
You've got multiple, you listened on the multiple logs, right? You're bringing your
-->
stored failed conversions, any failures there, any failures on the
-->
store, the JSON, all to have a log message. So everything looks good to me.
-->
I'll take silence as no, but do you have any questions?
-->
Right. Okay, Aaron, let's look at yours.
-->
If you can, can you go into your CSV to JSON?
-->
Awesome. Any issues? I see you have a file queued up.
-->
Did everything work well for you? Yeah.
-->
Perfect, perfect, perfect.
-->
All right.
-->
Cody, whose voice I should know. Very nice.
-->
If you want, you can log, you know,
-->
your log placement and your log conversion failure. You can reduce that to one
-->
processor and just log all your failures if you wanted.
-->
And then, you know, but you can,
-->
you know, you can have separate log strategies for separate processes.
-->
So, you know, totally your call. But I really like your labeling
-->
and the name mentions those types of things. So
-->
any questions? Let me know. Oh, Ben, you got some funnels going
-->
in. Wow.
-->
Outdoing yourself. So you even named the
-->
connections, you know, plus one for you.
-->
So that looks good. Hopefully you didn't have any issues. The only thing I would do is, you know,
-->
go through and add some labels. But, you know, the way you got this set up,
-->
I could easily read it. You may also want to consider, you know,
-->
how you align your processors. You know, you've seen already
-->
a lot of folks that will go straight down and go straight through the data flow.
-->
Or you can go left to right. If you speak Arabic, you can go right to left.
-->
You know, those types of things. So, you know, I just look at those types,
-->
you know, tips and tricks, then, you know, apply labels to make them understandable.
-->
But I don't see.