14 videos 📅 2024-05-06 08:00:00 America/Creston
39:11
2024-05-06 08:57:50
3:11:28
2024-05-06 10:30:19
24:39
2024-05-07 07:20:20
19:49
2024-05-07 08:03:25
1:14:35
2024-05-07 08:35:13
15:41
2024-05-07 10:06:25
38:33
2024-05-07 10:37:03
2:48
2024-05-07 11:19:01
59:37
2024-05-07 11:33:56
6:10
2024-05-07 14:55:25
39:40
2024-05-07 15:02:44
9:45
2024-05-08 06:44:21
29:27
2024-05-08 08:09:24
2:51:40
2024-05-08 12:09:24

Visit the Apache Nifi - GROUP 1 course recordings page

                WEBVTT

00:00:01.200 --> 00:00:11.320
All right, Randy, get CSVL, add the name after me, read input seasons, figure long, no, no.

00:00:11.700 --> 00:00:13.880
Randy, yours looks good as well.

00:00:14.720 --> 00:00:20.260
If you want to apply a label or any of those types of things, have at it, but it looks

00:00:20.260 --> 00:00:24.860
like yours is exactly like mine, and hopefully it worked.

00:00:24.920 --> 00:00:26.780
If it didn't, let me know if you have any questions.

00:00:26.780 --> 00:00:32.420
See, Brett, you added some color to your processors, like I did.

00:00:32.680 --> 00:00:33.460
Good job.

00:00:33.540 --> 00:00:34.680
I didn't explain that.

00:00:35.680 --> 00:00:37.680
So, you know, good job finding that out.

00:00:37.900 --> 00:00:44.340
You are now, you know, bending your arrows, you're beautifying this.

00:00:44.540 --> 00:00:45.860
I think it looks great.

00:00:46.460 --> 00:00:48.220
I really like the color scheme.

00:00:49.020 --> 00:00:51.940
You can go through and add labels like I did.

00:00:51.960 --> 00:00:54.840
If you wanted to, you could group these into a label.

00:00:55.840 --> 00:00:58.420
But overall, this looks great.

00:00:59.400 --> 00:01:00.800
If you have any questions, let me know.

00:01:02.100 --> 00:01:08.420
Amanda, Amanda, also the same, you know, added some colors to this.

00:01:09.460 --> 00:01:13.820
The only thing I do, maybe Amanda, is just like left to right.

00:01:14.480 --> 00:01:17.820
You have obtained a CSV down to update schema.

00:01:18.360 --> 00:01:22.680
You may just want to put these processors in order from left to right, and then,

00:01:22.680 --> 00:01:26.120
you know, go down to your log messages.

00:01:26.200 --> 00:01:31.960
So, you know, you keep your failures going left to right, and you keep your success

00:01:31.960 --> 00:01:38.740
left to right, and then use, you know, any downward flow as your log message.

00:01:39.120 --> 00:01:41.500
But it looks like you got it, you understand.

00:01:41.720 --> 00:01:44.320
So, you know, that's just any tips I would give.

00:01:46.040 --> 00:01:47.880
And finally, Alyssa.

00:01:48.120 --> 00:01:48.960
Perfect, perfect.

00:01:50.380 --> 00:01:52.220
You got naming.

00:01:52.500 --> 00:01:54.560
If you want, you can name your connections.

00:01:55.140 --> 00:01:59.500
You can apply some labels, you know, change color of your processors,

00:01:59.640 --> 00:02:00.820
those types of things.

00:02:01.540 --> 00:02:03.320
But yours looks good as well.

00:02:03.680 --> 00:02:05.440
Hopefully, you didn't have any issues.

00:02:05.520 --> 00:02:07.280
If you do, let me know.

00:02:07.800 --> 00:02:12.280
But no, great job, and it looks, you know, spot on.

00:02:13.460 --> 00:02:14.000
Okay.

00:02:14.920 --> 00:02:18.840
So, any final questions about your data flow?

00:02:19.680 --> 00:02:23.260
And again, you know, we used this to pick up a CSV file.

00:02:23.820 --> 00:02:31.040
We created a controller service, and, you know, then we used that service

00:02:31.040 --> 00:02:35.880
to do a CSV reader and have the data read in.

00:02:36.320 --> 00:02:41.360
We created an arrow schema that we just copied and pasted in, but, you know,

00:02:41.360 --> 00:02:45.360
if you were creating a schema, you could copy and paste your own schema in.

00:02:46.120 --> 00:02:51.080
And then we, you know, we used that schema to run the CSV files through

00:02:51.080 --> 00:02:57.260
to create a JSON document, push that back to the file system where we

00:02:57.260 --> 00:03:02.560
renamed it to a .json, even though it still had the .csv during lunch.

00:03:02.580 --> 00:03:07.520
I'm going to work on mine to come up with a regex pattern to really

00:03:07.520 --> 00:03:13.920
show that name off, you know, while I'm sitting here eating, and I'll paste in chat,

00:03:14.400 --> 00:03:18.160
you know, how to do that, you know, for those that want to update the,

00:03:18.160 --> 00:03:19.140
you know, their file name.

00:03:20.100 --> 00:03:25.240
So, any final questions before we move on to not file registry?

00:03:25.260 --> 00:03:25.740
Go ahead.

00:03:27.440 --> 00:03:32.580
So, on the get file, I noticed there's like a little symbol on it,

00:03:32.580 --> 00:03:34.960
it's like a shield, like a half red, half white shield.

00:03:34.960 --> 00:03:36.360
Did you cover what that meant yet?

00:03:38.260 --> 00:03:39.660
Let me see here.

00:03:42.220 --> 00:03:42.960
Right here?

00:03:47.240 --> 00:03:48.240
Is that the one you're talking about?

00:03:52.680 --> 00:03:56.700
Yeah, so there's that little shield, I'm just curious what that means.

00:03:59.240 --> 00:04:01.760
Let me see where I set that.

00:04:03.680 --> 00:04:04.960
It's on mine, too.

00:04:08.880 --> 00:04:11.240
So, there's the processor name shield.

00:04:14.560 --> 00:04:16.620
The red and white shield.

00:04:18.100 --> 00:04:22.120
The red and white shield, okay, I see this, you're talking about right here,

00:04:22.440 --> 00:04:25.180
this little icon here, right?

00:04:26.260 --> 00:04:26.560
Yes.

00:04:26.740 --> 00:04:27.420
Okay, sorry.

00:04:27.420 --> 00:04:32.760
So, that red and white shield, you know, in the UI indicates

00:04:32.760 --> 00:04:35.300
a restricted component processor.

00:04:36.880 --> 00:04:40.660
Meaning, these processors can be used to run unsanitized code

00:04:40.660 --> 00:04:43.140
or get data on a host.

00:04:43.580 --> 00:04:49.000
You know, it's just a quick visualization that, you know,

00:04:50.180 --> 00:04:54.300
there's, it's like risky, bingo, that's the word I'm looking for.

00:04:54.580 --> 00:04:57.220
So, if you noticed, it's on the get file

00:04:57.220 --> 00:04:58.460
and the put file.

00:04:59.920 --> 00:05:03.060
So, you know, the risk level is a little higher there.

00:05:04.420 --> 00:05:05.160
I figured that's what it was.

00:05:05.320 --> 00:05:07.500
So, if we were going to run like, I'm assuming there's like

00:05:07.500 --> 00:05:08.420
some sort of Python code.

00:05:08.440 --> 00:05:10.640
Would it also have that simple on it?

00:05:11.540 --> 00:05:12.420
It could.

00:05:13.040 --> 00:05:17.300
So, processors that are able to run code unchecked,

00:05:17.840 --> 00:05:20.120
you know, may have that as well.

00:05:22.140 --> 00:05:24.340
There's, so most likely, I think it does.

00:05:24.340 --> 00:05:28.240
Let me, actually, let's just look at it and see.

00:05:29.000 --> 00:05:33.420
We can drag the processor down to run Python code.

00:05:38.000 --> 00:05:38.900
Come on, wait and see.

00:05:47.060 --> 00:05:51.160
Yeah, you see the shield is right beside the script,

00:05:51.160 --> 00:05:56.480
validate record, invoke scripted processor, execute script,

00:05:57.040 --> 00:05:58.120
you know, those types of things.

00:05:58.420 --> 00:06:00.720
So, yeah, it is, yeah.

00:06:00.980 --> 00:06:02.660
It's like some sort of security.

00:06:03.020 --> 00:06:05.020
Yeah, it's a more risky thing.

00:06:05.340 --> 00:06:10.620
You can see, you know, where a lot of these

00:06:10.620 --> 00:06:14.420
is where it interacts with something outside of NAFA,

00:06:14.880 --> 00:06:16.960
right, because if it's in NAFA,

00:06:17.080 --> 00:06:20.140
if we're creating attributes, if we're dealing with data,

00:06:20.140 --> 00:06:23.580
you know, those types of things, you know,

00:06:23.620 --> 00:06:26.220
we, you know, it keeps it within NAFA

00:06:26.220 --> 00:06:28.960
or if there's like a secure connection,

00:06:29.780 --> 00:06:31.640
you know, those types of things,

00:06:32.520 --> 00:06:35.940
then we, you know, we won't have that shield.

00:06:35.980 --> 00:06:37.220
But yeah, that's a great,

00:06:37.320 --> 00:06:39.260
I've actually never been asked that question

00:06:39.260 --> 00:06:40.140
why there's a shield.

00:06:40.980 --> 00:06:44.820
But yeah, it's risky, you know,

00:06:44.820 --> 00:06:47.740
and it's a restricted component processor

00:06:47.740 --> 00:06:51.620
that can run unsanitized code, you know,

00:06:51.720 --> 00:06:54.320
or change or get data on the whole system,

00:06:55.200 --> 00:06:56.460
you know, those types of things.

00:06:56.920 --> 00:06:58.160
Great question though.

00:06:59.740 --> 00:07:04.780
Okay, any other questions, you know, about our flows?

00:07:06.600 --> 00:07:07.360
I had a question.

00:07:07.620 --> 00:07:08.060
Yes.

00:07:10.060 --> 00:07:12.620
So I know you mentioned that we have to create

00:07:12.620 --> 00:07:14.860
our own schema for the files.

00:07:15.420 --> 00:07:15.760
Yep.

00:07:15.760 --> 00:07:16.860
Okay.

00:07:16.860 --> 00:07:21.820
Well, what if, for example, in our office,

00:07:22.080 --> 00:07:24.180
we have a bunch of CSV files.

00:07:24.580 --> 00:07:28.320
Well, they don't follow like a very straightforward

00:07:28.840 --> 00:07:32.460
like a column or separated by values format.

00:07:33.360 --> 00:07:38.320
How can we come up with like a custom schema?

00:07:40.360 --> 00:07:43.800
So what I like to do on those cases

00:07:43.800 --> 00:07:47.880
is I like to read the CSV in,

00:07:48.980 --> 00:07:51.120
and you know, you can actually use,

00:07:53.180 --> 00:07:58.760
you can actually do like a lookup on that CSV as well.

00:07:59.100 --> 00:08:02.140
You can do a regex pattern.

00:08:03.080 --> 00:08:04.300
You know, so in the past,

00:08:04.340 --> 00:08:07.340
what I've done is used like a regex pattern

00:08:07.340 --> 00:08:11.360
to extract the data out of the CSV files.

00:08:11.360 --> 00:08:15.200
And I will take that data, extract it,

00:08:15.500 --> 00:08:17.660
and push it as an attribute.

00:08:18.620 --> 00:08:22.460
So if you look, you know, for JSON, for instance,

00:08:24.660 --> 00:08:28.580
you know, I can do, I can read that JSON,

00:08:28.720 --> 00:08:30.820
evaluate JSON path, for instance.

00:08:31.520 --> 00:08:33.440
You know, I can read that JSON that we,

00:08:34.180 --> 00:08:35.520
we, you know, wrote.

00:08:36.420 --> 00:08:40.440
I could change this to a flow file attribute

00:08:41.320 --> 00:08:44.180
and now I can read that JSON in

00:08:44.180 --> 00:08:47.300
and take all the fields and I can go through

00:08:47.300 --> 00:08:49.860
and start defining the fields, adding properties,

00:08:50.280 --> 00:08:53.100
you know, regex patterns or other things

00:08:53.100 --> 00:08:56.280
to pull the data out of a JSON

00:08:56.280 --> 00:08:59.100
and create an attribute with it.

00:08:59.360 --> 00:09:01.000
And then once it's an attribute,

00:09:01.380 --> 00:09:03.200
I can, you know, match,

00:09:03.340 --> 00:09:05.240
I can do a lookup if I needed to.

00:09:05.360 --> 00:09:08.220
I can match it to some other data if I needed to.

00:09:08.220 --> 00:09:12.420
And then I could, you know, extract that data

00:09:12.420 --> 00:09:15.000
even from a CSV, pull that data in,

00:09:15.200 --> 00:09:16.960
have it as an attribute, and then turn around

00:09:16.960 --> 00:09:21.500
and write that as proper JSON or proper CSV.

00:09:22.720 --> 00:09:26.560
So there's a couple of different ways to handle,

00:09:26.560 --> 00:09:28.900
you know, a mix of data.

00:09:30.660 --> 00:09:33.000
But yeah, it's, you're looking,

00:09:33.420 --> 00:09:35.540
if it's a lot of different formats

00:09:35.540 --> 00:09:36.780
and a lot of different things,

00:09:36.780 --> 00:09:39.120
you are now looking at, you know,

00:09:39.200 --> 00:09:42.360
doing regex patterns to extract that data

00:09:44.060 --> 00:09:47.200
or using the CSV reader

00:09:48.160 --> 00:09:49.860
and that controller service.

00:09:50.080 --> 00:09:51.180
So you may want, you know,

00:09:51.200 --> 00:09:52.980
if you remember in the CSV,

00:09:53.900 --> 00:09:56.040
that's a, this is a really good question,

00:09:56.040 --> 00:09:58.200
so let's go into that.

00:09:59.920 --> 00:10:00.860
In the CSV reader,

00:10:03.960 --> 00:10:08.660
we had like, you know, what is the record separator,

00:10:08.760 --> 00:10:13.140
the value separator, those types of things, right?

00:10:13.800 --> 00:10:18.760
So you could actually, depending on the content,

00:10:18.820 --> 00:10:21.380
you could read the contents of a CSV file,

00:10:21.820 --> 00:10:24.500
determine the, you know, the separator

00:10:24.500 --> 00:10:28.040
or something else that distinguishes it as different

00:10:28.040 --> 00:10:30.960
and send it to its own CSV reader.

00:10:30.960 --> 00:10:33.660
Where you have some of these,

00:10:33.820 --> 00:10:36.180
like the quote character may be different.

00:10:36.540 --> 00:10:40.120
Instead of a CSV, it may be a tab, a TSV.

00:10:40.400 --> 00:10:44.400
So, you know, you may have a tab in here.

00:10:44.720 --> 00:10:46.840
So you may have a few different

00:10:46.840 --> 00:10:49.160
CSV reader controller services

00:10:49.160 --> 00:10:53.320
and you will read the contents of that message

00:10:53.320 --> 00:10:55.700
or that piece of data

00:10:55.700 --> 00:10:59.980
and filter and route, you know,

00:10:59.980 --> 00:11:01.940
to the correct CSV reader.

00:11:03.560 --> 00:11:05.360
Does that help answer your question?

00:11:06.080 --> 00:11:08.060
If you have any like,

00:11:09.180 --> 00:11:11.160
like a better example of the data itself,

00:11:11.220 --> 00:11:15.180
I can help you like quickly just on a flow even.

00:11:15.720 --> 00:11:17.800
So, go ahead.

00:11:18.980 --> 00:11:21.140
Yeah, yeah, that definitely helps with that.

00:11:21.320 --> 00:11:23.620
I mean, I can show you what I have,

00:11:23.640 --> 00:11:25.040
but I don't think it's pretty.

00:11:25.460 --> 00:11:26.520
No, no.

00:11:28.760 --> 00:11:30.440
But another quick thing,

00:11:31.380 --> 00:11:34.340
is there any way to, since I'm a programmer,

00:11:34.720 --> 00:11:36.560
so I think I like to code,

00:11:36.680 --> 00:11:39.880
is there any like ways we can embed any code

00:11:39.880 --> 00:11:41.300
in like a data set or not?

00:11:42.860 --> 00:11:44.360
So, do you like Python?

00:11:46.060 --> 00:11:49.760
I mean, I know everyone works with Python.

00:11:49.940 --> 00:11:51.540
I haven't worked in a long time,

00:11:51.600 --> 00:11:53.320
but I mean, I'm trying to figure it out.

00:11:53.320 --> 00:11:55.420
You could

00:12:00.200 --> 00:12:02.400
do an execute script.

00:12:05.360 --> 00:12:06.520
Oh, come on, latency.

00:12:07.740 --> 00:12:09.480
You can do an execute script

00:12:09.480 --> 00:12:14.580
where you send that data and execute a script on it.

00:12:15.440 --> 00:12:17.080
You know, they're starting to get away

00:12:17.080 --> 00:12:19.420
with some of the scripting language

00:12:20.220 --> 00:12:22.860
that's used just because of security risks.

00:12:22.860 --> 00:12:26.220
But, you know, you could do

00:12:26.220 --> 00:12:27.940
Closure, for instance, or Groovy.

00:12:29.040 --> 00:12:30.240
In this processor,

00:12:30.920 --> 00:12:31.840
we have some other,

00:12:32.020 --> 00:12:33.740
and you can still do Python.

00:12:34.360 --> 00:12:35.340
It's just, you know,

00:12:35.360 --> 00:12:36.260
they're letting you know

00:12:36.260 --> 00:12:37.620
they're getting rid of that.

00:12:39.440 --> 00:12:40.940
We also have...

00:12:41.660 --> 00:12:42.560
Are they getting rid of Python?

00:12:43.060 --> 00:12:43.820
No, they're not.

00:12:43.860 --> 00:12:45.060
No, no, no, not at all.

00:12:45.180 --> 00:12:46.560
Actually, they're getting rid of it

00:12:46.560 --> 00:12:47.740
through that processor.

00:12:48.460 --> 00:12:52.120
The newest version of 9.5, 9.5 2.0,

00:12:52.120 --> 00:12:55.040
you can actually create Python processors

00:12:55.040 --> 00:12:57.380
because, remember, you know,

00:12:57.380 --> 00:12:59.980
yesterday we went over what a processor is

00:12:59.980 --> 00:13:03.120
and, you know, basically it's a Java jar.

00:13:04.280 --> 00:13:05.800
And so, you know,

00:13:05.800 --> 00:13:08.600
instead of creating a Java jar,

00:13:08.800 --> 00:13:11.000
Java nar for this instance

00:13:11.000 --> 00:13:13.300
with your own custom Java logic,

00:13:13.600 --> 00:13:16.060
you can actually create a Python processor

00:13:16.060 --> 00:13:18.440
to accept that incoming data.

00:13:18.640 --> 00:13:20.400
And then you may have, like,

00:13:20.400 --> 00:13:24.180
in Python a script to parse that

00:13:24.180 --> 00:13:26.300
and push that out as an attribute,

00:13:26.720 --> 00:13:27.360
for instance.

00:13:27.740 --> 00:13:29.240
You do have that capability.

00:13:29.880 --> 00:13:31.480
You also have...

00:13:35.340 --> 00:13:38.640
You can invoke a scripted processor.

00:13:39.140 --> 00:13:42.840
You can script a filter record.

00:13:43.780 --> 00:13:45.120
There's a few different ways

00:13:45.120 --> 00:13:48.320
where you can actually then use code,

00:13:48.320 --> 00:13:50.800
you know, to do these types of things.

00:13:50.920 --> 00:13:52.860
Even a scripted partition record,

00:13:53.380 --> 00:13:54.360
those types of things.

00:13:55.180 --> 00:13:57.480
So you can run your code,

00:13:57.600 --> 00:14:00.540
your custom code, you know, in this.

00:14:00.840 --> 00:14:02.040
You can invoke scripts.

00:14:02.440 --> 00:14:04.320
You can, you know,

00:14:04.400 --> 00:14:05.880
do those types of things if you'd like.

00:14:07.340 --> 00:14:07.640
Awesome.

00:14:08.140 --> 00:14:10.860
And if you get a free second,

00:14:11.020 --> 00:14:13.120
just chat with me, chat, like,

00:14:13.180 --> 00:14:15.500
a couple of CSV examples you're talking about

00:14:16.080 --> 00:14:17.780
and, like, later tonight

00:14:17.780 --> 00:14:19.480
I can whip up a quick...

00:14:19.480 --> 00:14:21.720
I'm not going to do the flow front to back,

00:14:22.300 --> 00:14:25.100
but I can show you an example

00:14:25.100 --> 00:14:26.280
of how I would do it

00:14:26.280 --> 00:14:27.940
using best design principles.

00:14:29.580 --> 00:14:31.060
Yeah, I mean, I can give you...

00:14:31.060 --> 00:14:33.460
Because we mostly deal with financial data

00:14:34.560 --> 00:14:37.140
and the way that it comes from...

00:14:37.140 --> 00:14:38.380
It comes from a website

00:14:38.380 --> 00:14:40.620
that we have to manually download the files.

00:14:42.000 --> 00:14:44.840
So it comes with a lot of extra tags and stuff.

00:14:44.840 --> 00:14:46.420
We have to strip up a lot of that.

00:14:46.420 --> 00:14:49.800
If you noticed in the downloads folder,

00:14:50.400 --> 00:14:52.140
one of my other hands-on

00:14:52.140 --> 00:14:54.040
is actually pulling from a website.

00:14:55.380 --> 00:14:57.000
How do you download that?

00:14:57.140 --> 00:14:58.500
You manually download it, you said.

00:14:58.820 --> 00:15:01.760
Is it like a zip file every day or what?

00:15:03.040 --> 00:15:03.880
It's a combination.

00:15:04.540 --> 00:15:05.360
It's a long process,

00:15:05.380 --> 00:15:08.240
but it comes in different CSV files

00:15:08.240 --> 00:15:10.120
or some of them have to be converted

00:15:10.120 --> 00:15:11.080
into cell files.

00:15:11.520 --> 00:15:12.100
Oh, okay.

00:15:13.060 --> 00:15:15.940
I was thinking if it's like an API you hit

00:15:15.940 --> 00:15:19.020
and it will give you the file, you could...

00:15:20.020 --> 00:15:22.260
Charlie, we've been asking for that

00:15:22.260 --> 00:15:23.500
for many years now.

00:15:24.080 --> 00:15:24.740
It's not the thing that happened.

00:15:25.100 --> 00:15:25.680
Oh, okay.

00:15:26.580 --> 00:15:30.780
Yeah, so it sounds like the source of the data

00:15:30.780 --> 00:15:34.180
isn't as automated as you would hope,

00:15:34.620 --> 00:15:38.320
but if they do get to that point,

00:15:38.840 --> 00:15:43.720
just remember we have HTTP processors as well.

00:15:45.000 --> 00:15:46.800
So you can actually get...

00:15:47.320 --> 00:15:48.800
You can handle HTTP requests.

00:15:49.900 --> 00:15:52.020
You can invoke HTTP.

00:15:52.440 --> 00:15:54.640
You can actually listen or post.

00:15:55.120 --> 00:15:57.520
There's a lot of capabilities there.

00:15:57.920 --> 00:15:59.660
So just for FYI,

00:15:59.920 --> 00:16:02.600
if your source gets to that capability

00:16:02.600 --> 00:16:06.460
where you can automatically pull,

00:16:06.880 --> 00:16:10.000
you may set up a dataflow to pull once a day

00:16:10.000 --> 00:16:12.800
and then you can take their output,

00:16:12.980 --> 00:16:17.040
filter, sort, parse it out, put it back together,

00:16:17.380 --> 00:16:19.940
and then you even have the capability

00:16:19.940 --> 00:16:21.780
to export as Excel.

00:16:25.660 --> 00:16:26.060
How about...

00:16:26.600 --> 00:16:28.240
I'm sorry, I don't mean to take over.

00:16:29.140 --> 00:16:30.960
Any SQL, MS SQL?

00:16:31.360 --> 00:16:34.260
We work with the SQL a lot.

00:16:35.360 --> 00:16:39.360
Yeah, so mostly if you work with like

00:16:41.040 --> 00:16:42.740
MS SQL, for instance,

00:16:42.760 --> 00:16:45.680
and I know you have access and stuff like that,

00:16:46.080 --> 00:16:51.880
but you have tons of put SQL query database table,

00:16:52.100 --> 00:16:55.000
query database table record, query record.

00:16:55.980 --> 00:16:58.060
You can list the database tables.

00:16:59.000 --> 00:17:02.200
You have tons of SQL capabilities.

00:17:07.300 --> 00:17:09.300
Does that help, Pedro?

00:17:12.960 --> 00:17:13.900
Yes, sir, I appreciate it.

00:17:13.960 --> 00:17:15.180
Okay, perfect, perfect.

00:17:15.640 --> 00:17:18.620
All right, any other final questions before we move on?

00:17:21.260 --> 00:17:22.800
Awesome, awesome, awesome.

00:17:23.280 --> 00:17:25.480
All right, so the next thing we're going to do

00:17:25.480 --> 00:17:29.200
is we've now built a couple of dataflows.

00:17:29.780 --> 00:17:32.180
You know, we do not want to take it too far.

00:17:32.180 --> 00:17:35.420
We're going to take a chance on turning a flow on

00:17:35.420 --> 00:17:39.920
and basically ingesting our own non-fi instance

00:17:39.920 --> 00:17:41.700
and breaking it.

00:17:41.980 --> 00:17:44.420
Maybe this crashes or something else like that,

00:17:44.420 --> 00:17:51.400
so let's start talking registry and non-fi registry.

00:17:52.420 --> 00:17:53.740
So if you can,

00:17:54.420 --> 00:17:56.320
you want to go back into your folder.

00:17:57.040 --> 00:17:58.540
You want to go back to your downloads.

00:18:00.840 --> 00:18:05.680
And we are going to install non-fi registry.

00:18:06.560 --> 00:18:14.580
So to do that, you want to go to the non-fi registry bin.zip

00:18:14.580 --> 00:18:16.140
which is right here.

00:18:16.600 --> 00:18:19.160
Go ahead and extract that,

00:18:19.300 --> 00:18:21.160
but don't run it yet if you don't mind.

00:18:22.380 --> 00:18:24.220
So extract it into the folder.

00:18:24.740 --> 00:18:26.420
You should see non-fi registry,

00:18:26.700 --> 00:18:28.520
you know, just like you were doing with non-fi.

00:18:28.520 --> 00:18:31.440
Give you just a minute on that.

00:18:31.800 --> 00:18:32.180
Okay.

00:18:32.960 --> 00:18:35.300
So you should have that extracted.

00:18:35.660 --> 00:18:37.920
We want to go into the non-fi registry folder.

00:18:38.800 --> 00:18:42.080
I have already executed mine,

00:18:42.820 --> 00:18:47.760
but, you know, again, it's the same principle with non-fi.

00:18:48.140 --> 00:18:52.220
You're going to have some already set folders

00:18:52.220 --> 00:18:54.880
and then when we start this,

00:18:55.140 --> 00:18:57.920
it will create the rest of the folders that you need.

00:18:57.920 --> 00:19:01.260
But the main folder that we are concerned about

00:19:01.260 --> 00:19:04.320
with registry is the comp folder.

00:19:04.840 --> 00:19:08.360
And so if you can, you know, go into your comp folder

00:19:08.360 --> 00:19:10.900
and then look at non-fi registry.

00:19:13.920 --> 00:19:18.520
So registry is pretty easy compared to non-fi.

00:19:18.560 --> 00:19:19.720
Some of the properties are,

00:19:19.720 --> 00:19:21.840
it's a lot less properties, thankfully,

00:19:22.360 --> 00:19:24.680
a little bit easier to configure,

00:19:25.360 --> 00:19:28.300
you know, but we'll kind of go through

00:19:28.300 --> 00:19:32.040
some of the more advanced configuration options.

00:19:32.440 --> 00:19:34.760
Your web properties, you should be able to see that.

00:19:35.820 --> 00:19:40.440
You know, we are basically listening on 0.0.0.0.

00:19:41.560 --> 00:19:44.080
You know, so it's going to bind to all the ports

00:19:45.000 --> 00:19:46.900
or bind to all the hosts.

00:19:47.420 --> 00:19:50.920
And then also the port is 18.080.

00:19:51.680 --> 00:19:53.300
You can change that here.

00:19:53.300 --> 00:19:54.660
I would just leave it the same

00:19:54.660 --> 00:19:56.420
because mine's set up for that

00:19:56.420 --> 00:19:59.680
and we're going to work off of that.

00:20:00.620 --> 00:20:02.580
How many threads the Jetty server should have

00:20:02.580 --> 00:20:06.500
for some of these more advanced like sysadmin type work?

00:20:07.480 --> 00:20:09.300
You know, you may want to mess with those.

00:20:11.720 --> 00:20:13.580
There's a lot of different tuning

00:20:13.580 --> 00:20:16.360
and performance considerations to take in

00:20:16.360 --> 00:20:19.640
if you're building this out in a more scalable

00:20:19.640 --> 00:20:21.440
excuse me, production way.

00:20:22.800 --> 00:20:24.680
Security properties, you know,

00:20:24.680 --> 00:20:25.820
here's where you're going to go in.

00:20:25.820 --> 00:20:27.540
You're going to start setting your key store,

00:20:27.660 --> 00:20:28.720
your trust store.

00:20:29.700 --> 00:20:32.480
You know, you can use the authorizers.xml

00:20:32.480 --> 00:20:35.000
that you may have created with Nafa.

00:20:36.380 --> 00:20:39.260
And so, you know, you can reference those.

00:20:39.680 --> 00:20:43.980
We will for this scenario where we do not have,

00:20:44.520 --> 00:20:47.660
you know, security enabled for the registry.

00:20:47.660 --> 00:20:51.920
You know, it's pretty much wide open

00:20:53.220 --> 00:20:56.560
but like Nafa is just a single sign on as well.

00:20:57.520 --> 00:21:00.360
So, you know, there's a lot there you can configure.

00:21:02.500 --> 00:21:06.740
And then the providers, you know,

00:21:06.740 --> 00:21:08.700
this is where you will go in.

00:21:09.700 --> 00:21:10.740
And so with registry,

00:21:11.720 --> 00:21:15.420
registry is backed by your versioning control.

00:21:15.420 --> 00:21:20.480
I mean, what do you all use for versioning control?

00:21:21.060 --> 00:21:22.420
Do you GitLab, GitHub?

00:21:23.620 --> 00:21:25.740
Azure has a Git service, I think.

00:21:26.200 --> 00:21:28.880
Are you, anyone able to tell me or is it?

00:21:29.940 --> 00:21:31.320
We use just two DevOps.

00:21:31.880 --> 00:21:32.740
Azure DevOps, okay.

00:21:35.720 --> 00:21:39.480
Okay, so there is a,

00:21:39.760 --> 00:21:41.420
let me take note of that.

00:21:43.660 --> 00:21:46.520
You can use Azure DevOps

00:21:47.640 --> 00:21:49.480
and I'll take note of that.

00:21:49.500 --> 00:21:52.940
I haven't configured Nafa registry to use that

00:21:52.940 --> 00:21:55.400
but I'm taking a note because I want to make sure

00:21:55.400 --> 00:21:59.680
I send you the right documentation for that.

00:22:00.260 --> 00:22:01.820
So, in your real world environment.

00:22:03.440 --> 00:22:04.020
It does.

00:22:07.240 --> 00:22:07.760
Okay, yeah.

00:22:07.760 --> 00:22:10.680
Yep, so perfect, perfect, perfect.

00:22:11.480 --> 00:22:14.220
Yeah, no, so under the hood it uses Git

00:22:14.220 --> 00:22:19.660
but we need to specify some things

00:22:19.660 --> 00:22:22.880
in that, you know, for Azure.

00:22:24.040 --> 00:22:27.400
So, and then you can use GitLab and GitHub as well.

00:22:28.000 --> 00:22:30.120
You know, the beauty of using Azure

00:22:30.120 --> 00:22:32.240
and this isn't really well known yet.

00:22:33.420 --> 00:22:37.320
So, Microsoft is incorporating Nafa

00:22:37.320 --> 00:22:39.920
more and more into its Azure stack.

00:22:42.120 --> 00:22:46.320
And, you know, I don't have 100% confirmation

00:22:46.320 --> 00:22:49.080
but, you know, Microsoft has been reaching out

00:22:49.080 --> 00:22:52.620
to some of us because they plan to make it

00:22:52.620 --> 00:22:55.380
as a service, as part of Azure.

00:22:55.980 --> 00:22:58.440
And so, you know, you will potentially

00:22:58.440 --> 00:23:00.960
in the future be able to configure this

00:23:00.960 --> 00:23:03.760
a little bit easier just because it's going to be

00:23:03.760 --> 00:23:06.500
officially supported as the Azure stack,

00:23:06.700 --> 00:23:08.120
if that makes sense.

00:23:09.800 --> 00:23:14.560
So, anyway, so, yeah, in the provider's file.

00:23:15.340 --> 00:23:17.420
In case it's relevant to your notes,

00:23:17.560 --> 00:23:19.100
we have an on-prem Azure DevOps,

00:23:19.220 --> 00:23:20.020
not the cloud version.

00:23:20.520 --> 00:23:21.160
There are differences.

00:23:21.540 --> 00:23:23.000
I don't know if you're...

00:23:23.000 --> 00:23:23.800
Since you were writing that down,

00:23:23.800 --> 00:23:24.700
I just figured...

00:23:24.700 --> 00:23:25.680
No, thank you.

00:23:25.900 --> 00:23:28.100
No, and like what I like to do

00:23:28.100 --> 00:23:29.920
is just get as much information as I can

00:23:29.920 --> 00:23:31.340
because at the end of this class,

00:23:31.340 --> 00:23:35.440
I want to make sure that I send all these notes out

00:23:35.440 --> 00:23:40.460
and as well as some more advanced capabilities

00:23:40.460 --> 00:23:43.060
and give you like a little handbook to work off of.

00:23:43.260 --> 00:23:46.040
And if I know the environment you're working off of,

00:23:46.220 --> 00:23:49.980
I can tailor that so when you get the email,

00:23:50.460 --> 00:23:51.280
we can go from there.

00:23:51.400 --> 00:23:53.400
So, it's Azure DevOps on-prem.

00:23:56.940 --> 00:23:57.500
Okay.

00:23:58.620 --> 00:24:02.800
So, you know, you'll actually go into your providers,

00:24:03.040 --> 00:24:05.780
into your conf directory to do that.

00:24:06.060 --> 00:24:08.260
If you look, you will see a providers.

00:24:13.820 --> 00:24:15.420
This one, for instance,

00:24:16.260 --> 00:24:18.600
you know, everything is commented out,

00:24:18.600 --> 00:24:22.380
but if we, you know, you would just use Git,

00:24:23.920 --> 00:24:26.720
access user, the password,

00:24:27.320 --> 00:24:29.360
the repository to clone,

00:24:30.420 --> 00:24:31.600
and those types of things.

00:24:32.400 --> 00:24:37.540
Like I said, I do not have a GitHub set up for this,

00:24:37.760 --> 00:24:41.200
but what I will do is go through

00:24:41.200 --> 00:24:43.340
and give you some directions

00:24:43.340 --> 00:24:46.140
on setting your GitHub up for this

00:24:46.140 --> 00:24:49.200
or GitLab or Azure DevOps

00:24:49.200 --> 00:24:51.660
and, you know, it will help you

00:24:51.660 --> 00:24:53.220
when you're configuring this in the future.

00:24:53.620 --> 00:24:57.040
We are worried about right now is non-fi to registry,

00:24:58.040 --> 00:24:59.720
but like I said, I'll give you instructions

00:24:59.720 --> 00:25:02.920
on registry to your Git version control.

00:25:03.580 --> 00:25:06.040
But you will define that in your providers,

00:25:06.380 --> 00:25:09.780
you know, just like the properties say.

00:25:10.540 --> 00:25:12.940
Any database properties under the hood,

00:25:13.800 --> 00:25:15.420
registry, non-fi,

00:25:15.420 --> 00:25:16.960
they like to use the H2,

00:25:17.700 --> 00:25:19.700
that's constantly being updated.

00:25:21.840 --> 00:25:25.640
Any extensions directory as well for AWS,

00:25:26.820 --> 00:25:28.640
there's some special configuration there.

00:25:29.460 --> 00:25:31.760
Identity mapping, you know,

00:25:31.800 --> 00:25:33.900
there's some additional security,

00:25:34.820 --> 00:25:36.660
Kerberos properties, you know,

00:25:36.720 --> 00:25:37.680
those types of things.

00:25:38.080 --> 00:25:40.500
So, you know, when you start putting this

00:25:40.500 --> 00:25:42.000
more in production,

00:25:42.580 --> 00:25:45.360
you're going to look at your conf directory first,

00:25:45.360 --> 00:25:47.000
start getting that filled out,

00:25:47.320 --> 00:25:50.320
and go to your providers and get that checked.

00:25:50.600 --> 00:25:53.460
What I will do is I will try to get you

00:25:53.460 --> 00:25:56.180
a good example of your providers

00:25:56.180 --> 00:25:58.120
so you can use your Azure DevOps,

00:25:59.100 --> 00:26:02.160
but, you know, this is where you will define it.

00:26:02.840 --> 00:26:05.300
So, with that being said, we have...

00:26:05.300 --> 00:26:08.700
Can I ask, what do we get out of connecting it to DevOps?

00:26:09.340 --> 00:26:10.900
Because I get the value of a registry,

00:26:11.600 --> 00:26:13.420
but if we just don't connect it to DevOps,

00:26:13.420 --> 00:26:15.100
what would be the difference?

00:26:16.540 --> 00:26:18.980
So, the way this process works is

00:26:18.980 --> 00:26:20.780
NonFi communicates with registry

00:26:20.780 --> 00:26:25.160
to store and version, you know,

00:26:25.400 --> 00:26:26.480
all of their data flows.

00:26:27.780 --> 00:26:30.400
But it's not backed,

00:26:30.580 --> 00:26:33.280
registry is not backed by Git

00:26:33.280 --> 00:26:35.700
or a true version control system,

00:26:36.080 --> 00:26:38.760
and so you would plug your registry

00:26:38.760 --> 00:26:40.580
into GitHub, for instance,

00:26:40.960 --> 00:26:43.400
and so when those flows are committed,

00:26:43.400 --> 00:26:45.820
you know, registry will take those,

00:26:45.940 --> 00:26:48.240
you know, create the history file

00:26:48.240 --> 00:26:50.760
and then push as a Git push

00:26:50.760 --> 00:26:52.660
to your Git repo.

00:26:52.940 --> 00:26:54.740
So, that way you have,

00:26:55.460 --> 00:26:57.540
you know, you may decide that

00:26:57.540 --> 00:26:59.700
as part of your CICD process,

00:27:00.000 --> 00:27:02.620
you will take a flow and push that out

00:27:02.620 --> 00:27:04.520
as well as the, you know,

00:27:04.620 --> 00:27:09.480
say you had an Ansible,

00:27:09.480 --> 00:27:12.440
you know, playbook set up to deploy NonFi,

00:27:12.440 --> 00:27:14.740
and you need to feed it

00:27:14.740 --> 00:27:16.920
the flow that it's going to use,

00:27:17.280 --> 00:27:18.800
and you can pull that

00:27:18.800 --> 00:27:20.660
from your Git repo like GitHub,

00:27:20.880 --> 00:27:21.300
for instance.

00:27:21.480 --> 00:27:22.120
Does that...

00:27:22.120 --> 00:27:23.420
Okay, so you could do that

00:27:23.420 --> 00:27:25.000
without connecting the registry

00:27:26.080 --> 00:27:27.260
to like a Git...

00:27:27.260 --> 00:27:28.240
Yes, yeah.

00:27:28.400 --> 00:27:30.580
Because we are going to save the versions

00:27:30.580 --> 00:27:31.660
into registry

00:27:32.620 --> 00:27:35.180
and not registry to GitHub.

00:27:35.800 --> 00:27:37.520
So, that final step,

00:27:37.660 --> 00:27:39.360
you know, would be in your environment

00:27:39.360 --> 00:27:41.520
you would have NonFi to registry

00:27:41.520 --> 00:27:44.940
and then registry to your Azure DevOps.

00:27:45.360 --> 00:27:46.620
Think of registry

00:27:46.620 --> 00:27:49.560
as your translation layer

00:27:49.560 --> 00:27:51.560
to get your data flows

00:27:51.560 --> 00:27:53.420
into a versioning control

00:27:53.420 --> 00:27:55.760
as well as a UI

00:27:55.760 --> 00:27:57.160
to manage your data.

00:27:57.360 --> 00:27:58.060
Okay, because I know

00:27:58.060 --> 00:27:59.880
I was speaking with James,

00:28:00.120 --> 00:28:01.420
he had, from what you see,

00:28:01.620 --> 00:28:02.320
he had been the one

00:28:02.320 --> 00:28:03.280
who has been messing with NonFi.

00:28:03.400 --> 00:28:04.380
He's not in this training, unfortunately.

00:28:05.100 --> 00:28:06.660
He developed a flow

00:28:06.660 --> 00:28:08.220
on a set production instance

00:28:08.220 --> 00:28:08.820
even though we don't have

00:28:08.820 --> 00:28:09.580
a real production instance,

00:28:09.620 --> 00:28:10.720
and then he exported it

00:28:10.720 --> 00:28:12.220
to like an XML or something

00:28:12.220 --> 00:28:13.700
and then had to convert it

00:28:13.700 --> 00:28:14.480
and then import it

00:28:14.480 --> 00:28:15.280
to another instance.

00:28:15.720 --> 00:28:17.080
So, this would be,

00:28:17.080 --> 00:28:18.080
this would be instead of that,

00:28:18.320 --> 00:28:19.020
tedious process.

00:28:20.000 --> 00:28:22.020
Yeah, so, yeah.

00:28:23.900 --> 00:28:25.340
And I'll show you

00:28:25.340 --> 00:28:26.760
when we set up registry,

00:28:27.880 --> 00:28:29.580
you know, the beauty is

00:28:29.580 --> 00:28:33.040
registry will segregate

00:28:33.040 --> 00:28:34.180
everything into buckets.

00:28:34.840 --> 00:28:37.440
And so, you can go in

00:28:37.440 --> 00:28:39.100
with permissions, for instance,

00:28:39.100 --> 00:28:40.600
say that, you know,

00:28:40.620 --> 00:28:42.580
user X only has access

00:28:42.580 --> 00:28:43.440
to these buckets,

00:28:43.660 --> 00:28:44.780
user Y only has access

00:28:44.780 --> 00:28:45.800
to these buckets,

00:28:46.680 --> 00:28:47.740
but, you know,

00:28:47.740 --> 00:28:49.100
if you have X and Y

00:28:49.100 --> 00:28:50.380
working on the same thing,

00:28:50.380 --> 00:28:52.640
they can actually, you know,

00:28:52.760 --> 00:28:53.920
commit to registry,

00:28:54.300 --> 00:28:55.680
and then user Y

00:28:55.680 --> 00:28:57.100
can check that out

00:28:57.100 --> 00:28:58.920
and pull the latest version

00:28:58.920 --> 00:29:00.640
and continue working

00:29:00.640 --> 00:29:01.760
on the flow as well.

00:29:02.060 --> 00:29:02.920
And then it's just,

00:29:02.920 --> 00:29:03.160
you know,

00:29:03.180 --> 00:29:04.780
the committee's back and forth

00:29:04.780 --> 00:29:06.520
and then registry is going

00:29:06.520 --> 00:29:07.900
to take that flow

00:29:07.900 --> 00:29:09.480
and push it to

00:29:09.480 --> 00:29:11.440
a versioning control system

00:29:11.440 --> 00:29:12.520
like GitHub

00:29:12.520 --> 00:29:15.160
that will keep track of it,

00:29:15.200 --> 00:29:15.560
you know,

00:29:15.560 --> 00:29:16.440
you can branch

00:29:16.440 --> 00:29:17.840
and all those things.

00:29:18.140 --> 00:29:19.080
And then you will,

00:29:19.080 --> 00:29:21.100
you can use that flow

00:29:21.100 --> 00:29:23.460
as part of a CICD process

00:29:23.460 --> 00:29:25.120
to push that flow out.

00:29:25.440 --> 00:29:25.680
You know,

00:29:25.680 --> 00:29:26.880
here's the version

00:29:26.880 --> 00:29:28.120
that you need to push out

00:29:28.120 --> 00:29:30.700
for dev or prod or whatever.

00:29:31.580 --> 00:29:33.000
And you can have that

00:29:33.000 --> 00:29:33.700
as part of your

00:29:33.700 --> 00:29:35.100
CICD process as well.

00:29:37.900 --> 00:29:39.100
Thank you.

00:29:39.200 --> 00:29:39.520
Yep, yep.

00:29:40.040 --> 00:29:41.360
Great questions

00:29:41.360 --> 00:29:42.460
and this is

00:29:42.460 --> 00:29:44.100
the perfect time to ask it.

00:29:44.160 --> 00:29:44.760
Okay.

00:29:45.340 --> 00:29:45.940
So,

00:29:46.220 --> 00:29:47.000
we now have

00:29:47.000 --> 00:29:48.500
all of our files extracted.

00:29:49.120 --> 00:29:49.860
We've looked at

00:29:49.860 --> 00:29:51.060
our comp directory.

00:29:52.020 --> 00:29:54.640
There is a database directory,

00:29:54.960 --> 00:29:55.180
you know,

00:29:55.240 --> 00:29:56.520
it's an H2 that's

00:29:56.520 --> 00:29:57.580
keeping track of things.

00:29:58.780 --> 00:29:59.060
You know,

00:29:59.060 --> 00:29:59.900
there is docs,

00:30:01.400 --> 00:30:03.400
every NiFi component

00:30:03.400 --> 00:30:04.600
except for like

00:30:04.600 --> 00:30:05.820
some of the minify stuff

00:30:05.820 --> 00:30:07.480
that we will go into later.

00:30:07.480 --> 00:30:08.660
You know,

00:30:08.800 --> 00:30:09.940
has docs built in

00:30:09.940 --> 00:30:10.660
just in case you're

00:30:10.660 --> 00:30:11.240
on a network

00:30:11.240 --> 00:30:12.500
that doesn't have access

00:30:12.500 --> 00:30:13.800
to the internet.

00:30:15.380 --> 00:30:16.000
Lib directory,

00:30:16.900 --> 00:30:17.140
you know,

00:30:17.200 --> 00:30:17.920
just like NiFi

00:30:17.920 --> 00:30:19.600
has that lib directory.

00:30:19.980 --> 00:30:21.520
There is an extensions directory

00:30:21.520 --> 00:30:22.280
if you want to build

00:30:22.280 --> 00:30:23.420
onto registry.

00:30:24.680 --> 00:30:24.980
You know,

00:30:25.160 --> 00:30:26.440
just like NiFi,

00:30:26.560 --> 00:30:27.200
you can build

00:30:27.200 --> 00:30:28.240
a processor

00:30:28.240 --> 00:30:29.020
and put it into

00:30:29.020 --> 00:30:29.720
the extensions

00:30:29.720 --> 00:30:30.720
and hot load it.

00:30:31.200 --> 00:30:32.060
There's some extensions

00:30:32.060 --> 00:30:32.680
you can build

00:30:32.680 --> 00:30:33.680
for registry.

00:30:34.440 --> 00:30:35.280
I don't know

00:30:35.280 --> 00:30:37.160
if I've ever seen one

00:30:37.160 --> 00:30:37.860
built,

00:30:38.600 --> 00:30:39.320
to be honest,

00:30:39.480 --> 00:30:41.280
but the capabilities there.

00:30:41.940 --> 00:30:42.640
And of course,

00:30:43.020 --> 00:30:44.100
your logs directory

00:30:44.100 --> 00:30:45.240
because that way you can

00:30:45.800 --> 00:30:47.120
see what's going on

00:30:47.120 --> 00:30:48.140
and those types of things.

00:30:48.920 --> 00:30:50.720
So for this exercise,

00:30:51.160 --> 00:30:51.960
let's go into

00:30:51.960 --> 00:30:53.120
the bin directory.

00:30:54.660 --> 00:30:56.140
And we want to run

00:30:56.140 --> 00:30:57.480
NiFi registry.

00:30:58.140 --> 00:31:00.880
And so that should bring up

00:31:00.880 --> 00:31:02.240
a new window

00:31:03.620 --> 00:31:04.260
that,

00:31:04.260 --> 00:31:05.140
you know,

00:31:05.140 --> 00:31:06.560
is our NiFi registry.

00:31:13.520 --> 00:31:15.940
Let me check your screen.

00:31:20.500 --> 00:31:22.600
And then once it starts,

00:31:22.860 --> 00:31:23.600
if you can,

00:31:23.660 --> 00:31:24.660
open your browser.

00:31:25.460 --> 00:31:27.240
If you want to go to,

00:31:29.360 --> 00:31:30.000
if you remember,

00:31:30.280 --> 00:31:31.340
we set our,

00:31:31.340 --> 00:31:31.640
you know,

00:31:31.640 --> 00:31:32.660
we didn't change our port,

00:31:32.720 --> 00:31:35.620
but it was 18.080.

00:31:35.620 --> 00:31:37.740
So if you can,

00:31:37.780 --> 00:31:38.720
when it's up and running,

00:31:38.800 --> 00:31:40.140
bring up a new tab

00:31:40.140 --> 00:31:41.680
and you want to log in

00:31:41.680 --> 00:31:45.580
and go to 127.0.0.1

00:31:45.580 --> 00:31:48.520
colon 18.080.

00:31:48.560 --> 00:31:49.100
Sorry, Joshua,

00:31:49.420 --> 00:31:50.920
I'm stuck for a little bit.

00:31:51.140 --> 00:31:52.580
Can I make that last step?

00:31:53.380 --> 00:31:54.080
Oh, no worries.

00:31:55.600 --> 00:31:58.180
So in your bin folder

00:31:58.180 --> 00:31:59.220
in NiFi registry,

00:31:59.420 --> 00:32:00.120
you've extracted

00:32:00.120 --> 00:32:01.060
NiFi registry,

00:32:01.180 --> 00:32:02.860
you've extracted that zip file.

00:32:03.200 --> 00:32:04.480
In the bin folder

00:32:04.480 --> 00:32:07.720
is the run NiFi registry bat file.

00:32:08.260 --> 00:32:09.640
And so you want to run that

00:32:09.640 --> 00:32:11.140
and it's going to bring up

00:32:11.140 --> 00:32:13.180
a new command line window

00:32:13.180 --> 00:32:13.720
that's running.

00:32:14.100 --> 00:32:15.320
So you should have two,

00:32:15.380 --> 00:32:16.300
one for NiFi,

00:32:16.400 --> 00:32:17.740
one for NiFi registry.

00:32:20.040 --> 00:32:22.280
And then once registry starts,

00:32:22.440 --> 00:32:22.800
you know,

00:32:22.800 --> 00:32:23.860
you got to give it a minute.

00:32:24.400 --> 00:32:26.560
You should be able to log in

00:32:27.100 --> 00:32:27.840
to the,

00:32:27.840 --> 00:32:28.660
to be able to log in

00:32:28.660 --> 00:32:29.180
to the,

00:32:29.180 --> 00:32:29.860
to the browser.

00:32:36.120 --> 00:32:41.300
Ben, where are you at?

00:32:41.400 --> 00:32:42.700
Ben, you blew up.

00:32:43.920 --> 00:32:45.340
Oh, I'm kind of like, hold on.

00:32:45.500 --> 00:32:47.080
I downloaded 2.0 and that wasn't that 45.

00:32:47.360 --> 00:32:48.400
Can you say it?

00:32:48.640 --> 00:32:50.400
Yeah, the one,

00:32:50.400 --> 00:32:52.080
2.5 should be,

00:32:52.300 --> 00:32:53.580
oh, you're re-downloading.

00:32:54.600 --> 00:32:55.180
Yeah, 45.

00:32:55.320 --> 00:32:57.860
So make sure you're downloading registry

00:32:57.860 --> 00:32:59.040
so scroll back up.

00:32:59.040 --> 00:32:59.240
So you're downloading

00:32:59.680 --> 00:33:01.860
and click on registry

00:33:03.840 --> 00:33:07.180
and you can use 126,

00:33:08.420 --> 00:33:09.100
binaries.

00:33:14.760 --> 00:33:16.040
I know, I know.

00:33:16.220 --> 00:33:17.400
If I could change it,

00:33:17.400 --> 00:33:18.140
you know, change this,

00:33:18.140 --> 00:33:19.100
trust me I would.

00:33:23.000 --> 00:33:23.520
Okay.

00:33:24.900 --> 00:33:26.540
So Ben's downloading

00:33:26.540 --> 00:33:29.940
the registry now.

00:33:30.240 --> 00:33:32.080
It should also be in your folder.

00:33:32.280 --> 00:33:33.640
If you don't have it,

00:33:33.640 --> 00:33:35.700
you can go to the NotFile website

00:33:35.700 --> 00:33:36.880
and download it.

00:33:37.200 --> 00:33:37.780
The reason,

00:33:38.300 --> 00:33:38.620
you know,

00:33:39.180 --> 00:33:42.260
I would tell this is the more technically advanced class.

00:33:43.180 --> 00:33:43.660
You know,

00:33:43.660 --> 00:33:46.500
if it wasn't the technically advanced class,

00:33:46.640 --> 00:33:48.640
I may have already have this installed

00:33:48.640 --> 00:33:49.880
and running for everyone.

00:33:51.060 --> 00:33:51.460
But,

00:33:51.780 --> 00:33:52.380
you know,

00:33:52.460 --> 00:33:55.340
so because we're the advanced class,

00:33:55.340 --> 00:33:56.640
we're going to go through

00:33:56.640 --> 00:33:58.360
some of the technical aspects of this,

00:33:58.400 --> 00:34:00.160
setting it up and things like that.

00:34:00.940 --> 00:34:01.440
So yeah,

00:34:01.440 --> 00:34:03.320
if you don't already have it downloaded,

00:34:03.420 --> 00:34:04.780
you can download it yourself.

00:34:05.280 --> 00:34:08.280
Again, all of this is free and open source.

00:34:09.260 --> 00:34:10.140
And we just have it.

00:34:10.140 --> 00:34:12.420
What was the URL for that new network?

00:34:13.980 --> 00:34:14.440
Yeah.

00:34:14.640 --> 00:34:16.840
So if you want to download it yourself,

00:34:16.920 --> 00:34:19.520
it should be available on your

00:34:19.520 --> 00:34:21.440
NPEG or I'll pull your screen up.

00:34:21.440 --> 00:34:24.520
And go to notfile.apache.org.

00:34:26.600 --> 00:34:28.000
Go to click download

00:34:29.140 --> 00:34:31.200
and you click on registry.

00:34:31.460 --> 00:34:32.300
Scroll back up.

00:34:33.520 --> 00:34:34.460
There's three tabs,

00:34:34.840 --> 00:34:36.200
NotFile, Minify, Registry.

00:34:36.340 --> 00:34:37.560
Well, four tabs and FBS

00:34:37.560 --> 00:34:38.720
is the Flow Design System.

00:34:39.080 --> 00:34:39.860
Click on registry.

00:34:39.900 --> 00:34:40.560
Go up,

00:34:40.560 --> 00:34:40.900
up top.

00:34:41.060 --> 00:34:41.340
There you go.

00:34:41.420 --> 00:34:41.680
Perfect.

00:34:42.060 --> 00:34:43.460
Scroll down a little bit.

00:34:44.000 --> 00:34:47.160
And you see the 125 registry binaries.

00:34:47.560 --> 00:34:48.460
Not the source,

00:34:48.600 --> 00:34:49.400
unless you want to build it.

00:34:49.400 --> 00:34:49.940
There you go.

00:34:51.120 --> 00:34:53.960
And it should have been available.

00:34:54.280 --> 00:34:55.740
I think you're good.

00:34:56.720 --> 00:34:58.280
You click the HTTP site.

00:34:58.360 --> 00:34:59.940
If it doesn't respond in a minute,

00:34:59.980 --> 00:35:01.440
click the backup site.

00:35:02.900 --> 00:35:05.260
But it should download those files.

00:35:07.200 --> 00:35:09.020
And what I was saying is

00:35:09.760 --> 00:35:11.180
the NotFile registry

00:35:11.180 --> 00:35:13.780
should it be already in your downloads folder?

00:35:15.300 --> 00:35:17.000
When I created the VMs,

00:35:17.020 --> 00:35:19.300
I created it with it all as a zip file.

00:35:19.300 --> 00:35:21.000
So you don't have to download it,

00:35:21.000 --> 00:35:23.780
but you may have picked all your zips up

00:35:23.780 --> 00:35:25.120
and destroyed it.

00:35:25.620 --> 00:35:27.560
I think we all messed up the zips.

00:35:28.200 --> 00:35:29.300
Ah, there we go.

00:35:29.300 --> 00:35:30.400
I had to re-download.

00:35:31.360 --> 00:35:32.920
No worries, no worries.

00:35:33.760 --> 00:35:34.200
Like I said,

00:35:34.200 --> 00:35:36.520
we use that zip flow file

00:35:36.520 --> 00:35:39.380
as an example of what bad can happen as well.

00:35:43.860 --> 00:35:46.000
So yeah, just download it.

00:35:46.080 --> 00:35:48.160
Once you've got that zip file,

00:35:48.160 --> 00:35:51.560
extract it and go into...

00:35:51.560 --> 00:35:53.820
We're not going to change anything in the properties,

00:35:54.700 --> 00:35:56.380
but then go into the bin file

00:35:56.380 --> 00:35:58.100
and click Run Registry.

00:35:58.900 --> 00:36:00.120
And I'll give everyone...

00:36:00.120 --> 00:36:03.240
I get the connection refused, so...

00:36:03.800 --> 00:36:03.940
Right.

00:36:04.260 --> 00:36:05.960
So I think I have that.

00:36:06.300 --> 00:36:08.780
So then what happens after I run the registry?

00:36:09.200 --> 00:36:10.680
Okay, so after you...

00:36:10.680 --> 00:36:11.860
So Pedro, it's running.

00:36:13.380 --> 00:36:14.240
I think so, yeah.

00:36:14.480 --> 00:36:16.080
Let me look at your screen.

00:36:16.420 --> 00:36:17.420
Yep, it's running.

00:36:17.420 --> 00:36:19.840
So now in your address bar,

00:36:20.240 --> 00:36:20.500
go to...

00:36:22.500 --> 00:36:25.500
Just type in 127.0.0.1

00:36:27.260 --> 00:36:29.880
and you want to go to colon

00:36:31.780 --> 00:36:33.380
18-080

00:36:33.380 --> 00:36:34.220
and hit Enter.

00:36:34.840 --> 00:36:38.200
Oh, in front of that, put NAFAI-Registry.

00:36:39.100 --> 00:36:40.520
Do a front slash

00:36:41.840 --> 00:36:43.200
NAFAI-Registry.

00:36:43.240 --> 00:36:44.100
NAFAI-Registry.

00:36:45.360 --> 00:36:45.800
Oh.

00:36:53.000 --> 00:36:53.600
Okay.

00:36:54.680 --> 00:36:56.500
So it should look like that.

00:36:56.520 --> 00:36:58.900
You don't need to use HTTPS or anything else.

00:36:59.000 --> 00:36:59.820
Okay, perfect, Pedro.

00:37:00.080 --> 00:37:01.500
You're in.

00:37:02.000 --> 00:37:02.920
In my clean.

00:37:05.160 --> 00:37:08.580
Okay, anyone else having an issue?

00:37:09.760 --> 00:37:10.960
Yeah, I am.

00:37:11.200 --> 00:37:11.260
Sean.

00:37:11.540 --> 00:37:12.200
Hey, Sean.

00:37:12.840 --> 00:37:13.300
Let's take a look.

00:37:13.460 --> 00:37:15.360
Mine's not detected.

00:37:16.100 --> 00:37:17.140
No worries.

00:37:17.140 --> 00:37:18.580
We'll see what we got.

00:37:18.580 --> 00:37:19.760
Got it running.

00:37:20.040 --> 00:37:20.440
Uh-huh.

00:37:21.360 --> 00:37:23.640
I think I'm doing something wrong here.

00:37:23.640 --> 00:37:26.720
Yeah, I think you have HTTPS.

00:37:27.060 --> 00:37:29.440
It's just HTTP because...

00:37:31.200 --> 00:37:31.680
Yeah, perfect.

00:37:31.780 --> 00:37:34.260
So take your HTTPS out.

00:37:34.440 --> 00:37:35.020
There you go.

00:37:35.160 --> 00:37:35.600
Hit Enter.

00:37:36.820 --> 00:37:37.700
And you're good.

00:37:37.840 --> 00:37:38.320
Okay.

00:37:39.600 --> 00:37:42.440
So NAFAI out of the box

00:37:42.440 --> 00:37:45.300
comes with a self-signed certificate

00:37:45.300 --> 00:37:46.940
with a username and password.

00:37:48.040 --> 00:37:49.740
And, you know, the reason being is

00:37:49.740 --> 00:37:54.020
is we had tons of people downloading it,

00:37:54.080 --> 00:37:55.400
putting it in AWS,

00:37:55.720 --> 00:37:57.400
leaving it wide open to the world.

00:37:57.780 --> 00:37:59.560
And I can create a data flow

00:37:59.560 --> 00:38:01.720
to download whatever kind of,

00:38:01.720 --> 00:38:03.500
you know, malicious activity I want

00:38:03.500 --> 00:38:05.600
and, you know, go from there.

00:38:06.180 --> 00:38:09.580
So Registry, because Registry's not actually, like,

00:38:09.820 --> 00:38:11.180
executing code or anything,

00:38:11.940 --> 00:38:14.240
they decided to just leave that open.

00:38:14.240 --> 00:38:16.880
But you do have the capability

00:38:16.880 --> 00:38:20.660
to configure security in the conf files.

00:38:21.320 --> 00:38:22.380
You know, you want to put, you know,

00:38:22.620 --> 00:38:24.840
modify your authorizers, those types of things.

00:38:26.060 --> 00:38:28.260
But you're where you need to be.

00:38:30.740 --> 00:38:32.520
So how's everybody else doing?