39:11
2024-05-06 08:57:50
3:11:28
2024-05-06 10:30:19
24:39
2024-05-07 07:20:20
19:49
2024-05-07 08:03:25
1:14:35
2024-05-07 08:35:13
15:41
2024-05-07 10:06:25
38:33
2024-05-07 10:37:03
2:48
2024-05-07 11:19:01
59:37
2024-05-07 11:33:56
6:10
2024-05-07 14:55:25
39:40
2024-05-07 15:02:44
9:45
2024-05-08 06:44:21
29:27
2024-05-08 08:09:24
2:51:40
2024-05-08 12:09:24
Visit the Apache Nifi - GROUP 1 course recordings page
WEBVTT
-->
So you can save your flow to the same bucket or you can create a new bucket and save your
-->
other flow to that one and I would be curious if someone would, I'm going to test this
-->
out maybe.
-->
I'm going to put, I'm going to see if I can add my registry to one of you all's
-->
NaPy instance.
-->
I don't know if the IP address will resolve in these VMs.
-->
I don't think they're sharing in between them, but I'd be curious to see.
-->
But anyway, so you should now have, you know, two flows within your, within your
-->
registry, everyone does.
-->
Okay, perfect.
-->
If you do not, let me know.
-->
So let's go back to NaPy and let's go into that CSV to JSON data flow.
-->
And what I want to do is I am going to make some changes to this.
-->
I want to add, you know, a label.
-->
I'm just adding a label, just making some changes to my data flow.
-->
Okay, so the change I made is basically I just added a label to this and if you go
-->
back, you know, you can use your break from trail to, to go back or just right
-->
click and say, leave your group, but when you go back, you know, that green check
-->
box is now, and so what that lets you know, what's that letting you know is
-->
that you have changes that need to be committed and so you can right click
-->
again, same version, and you want to commit local changes.
-->
And so, but this version comment, I want to put, I added a label and then once I
-->
make that comment, it should be a check box and I should be good to go.
-->
Hopefully, if anyone, I'm not looking at everyone's screen right now.
-->
So if anyone's having an issue, feel free to speak up if you're not this far along.
-->
Okay, so now that I've committed my changes and it's now into the registry,
-->
if I had a GitHub or GitLab or Azure DevOps platform set up, you know,
-->
Git repo there, you know, the registry would automatically write these data
-->
flows to that versioning control system and utilize that version control, you
-->
know, for storage of all these data flows.
-->
I can use them outside of registry, you know, those types of things, you know,
-->
I may, we're going to get into this when we do some Minify stuff, but
-->
Minify, you need to develop your flows in Non-Fi and save them before Minify
-->
could run them, because Minify does not have a UI to develop its flows.
-->
And that is the, you know, from tomorrow morning to lunch tomorrow,
-->
we're working Minify, you know, I've set aside half a day just for that
-->
because I know it's a very, there's a lot of things going on once we start
-->
introducing Minify, but, you know, that could be a reason you want to save your
-->
flow, you built it in Non-Fi, you save it, you save it to you,
-->
and it gets backed up to your Git repo.
-->
You may have a CI CD process that will push Non-Fi to your, you know,
-->
Minify to your Raspberry Pi Edge device, it will push the correct flow
-->
to that as well, because you do not have registry installed on that Pi,
-->
you do not have Non-Fi installed, you do not have registry, you know,
-->
connected to that Non-Fi that's not installed, so, you know,
-->
Minify is probably one of the key users of that virtual control system,
-->
and, you know, in talking with, you know, folks,
-->
Minify is definitely, you know, looked at for some of you all's projects,
-->
so, you know, just keep that in mind.
-->
Now, with that being said, I can stop versioning control at any time.
-->
I can stop, you know, if I had to, you know, I don't know of a good reason
-->
to stop versioning control, but if you have, you know, your data
-->
flow at a good state, you may want to stop, you know,
-->
versioning it, or if it's going away, or you may want to mess with it locally
-->
and make a lot of changes, and then once you get to where you're at,
-->
then start committing again.
-->
I can't think of any real business use cases to stop versioning control
-->
except for like testing and training and those types of things.
-->
You can also revert back.
-->
So, what I did is I'm going to change my version back to the original.
-->
So, if you remember, I went through and added a label,
-->
and what I want to do is revert back and make that change.
-->
So, it's stopping all the processors
-->
and then updating the data flow that's in there.
-->
So, when I go back into here, my label that I created is gone.
-->
And the red arrow lets me know that there is a newer version of this flow out there.
-->
So, I was reverting my changes because I wanted to get rid of that label,
-->
and maybe I wanted to, you know, get rid of that label,
-->
and I wanted to add another label.
-->
By the way, you can control C, control V on labels.
-->
Maybe I wanted two labels now, right?
-->
So, what I would do is I created my new label,
-->
I have changes that needed to be made,
-->
and now I can, you know, I can revert my local changes,
-->
I can show the local changes,
-->
or I can just commit a newer version.
-->
And now we are back, you know, to a green checkbox.
-->
Okay. So, with that being said, now that is how we are committing our changes,
-->
that's how we are working with our data flow, you know, those types of things.
-->
So, what I want to do is I am going to stop my processor group.
-->
I am going to delete that processor group.
-->
Well, I would have to go in and delete the controller service
-->
that's associated with it, you know, some of these other things.
-->
But just stop your processor group.
-->
And now we have a version of that data flow.
-->
So, now we can actually go and say,
-->
drag and drop down a new processor group.
-->
And now we have the option to import from registry.
-->
So, previously we did not have this.
-->
We could only create a name for our processor group
-->
or upload like we did previously for the data flow that we were working with.
-->
You know, that JSON element that we can upload.
-->
But now you can import from registry.
-->
You can go to your buckets, and if you have many flow names,
-->
you can select the bucket that your flow is in,
-->
select the, you know, the flow name and the version.
-->
So, for this one, I want to go back to my original version.
-->
I am going to import that.
-->
And done.
-->
So, what it's letting me know, again, that is a,
-->
there is a newer version of this available.
-->
I pulled in version one.
-->
I now have three different versions.
-->
And, you know, those types of things.
-->
So, what I can do, as well as I can stop versioning control
-->
this process group, for instance,
-->
I can start versioning control.
-->
And if I wanted to, I could push this to a new bucket.
-->
I can give it a new name.
-->
So, now I have, you know, two different flows.
-->
You may have more.
-->
But I have two different flows saved in my registry.
-->
Refresh.
-->
Awesome.
-->
So, in that same bucket, I now have, you know,
-->
a new name, data flow, and I can go back to my registry.
-->
I also have the CSV to JSON that has three different versions.
-->
I can refresh these.
-->
I can delete the bucket altogether.
-->
I can export a version.
-->
I can import even a new version.
-->
Or I can just delete the flow altogether.
-->
And then, you know, you can go back to your settings here
-->
and, you know, delete your buckets
-->
and things like that as well.
-->
You know, so NaPy registry has a, you know,
-->
a ton of capabilities in managing your data flows.
-->
And, you know, now, if you were to run your import zip
-->
and, you know, it blows up, it, you know,
-->
it ingests your parent NaPy directory.
-->
And, you know, it no longer works.
-->
We can now delete that whole instance, unzip it,
-->
start NaPy, add this registry real quickly,
-->
and now we have a version of the flows.
-->
You know, so there's a lot of capabilities here.
-->
Again, we're not going to go into configuring GitHub
-->
and GitLab and Azure and those types of things.
-->
But what I will do is send out some tips and tricks
-->
and instructions on how I've done this
-->
and some reference material because, you know,
-->
you may need to reference a specific class
-->
or property in the configuration file.
-->
So I'm going to pause there.
-->
I think that just about covers, you know, the registry.
-->
The user guide and the system administrator guide
-->
is online.
-->
Those links I will send out as well.
-->
You know, it goes into a little bit more detail
-->
about, you know, managing users and things like that.
-->
If you remember when I was talking about
-->
some of the NaPy security is very fine detailed.
-->
It's, you know, it's fine grained controls.
-->
You can, you know, set your users to only allow them
-->
to view a data flow.
-->
You can start a data flow.
-->
They can import just the data flow, things like that.
-->
And registry is set up the same.
-->
So, you know, you can set up policies
-->
where, you know, someone could come in
-->
and view the registry but not make any delete.
-->
You cannot upload those types of things.
-->
So for you sysadmins out there,
-->
you know, it's good to keep those things in mind
-->
when you're setting this up in a more secure environment
-->
because you want to enable those fine grained controls.
-->
So with all that being said, did anyone have any issues
-->
creating the registry, managing the bucket,
-->
you know, any of those types of things?
-->
We're good on our end.
-->
Awesome, awesome, awesome.
-->
I didn't have any issues but I have some questions.
-->
Yeah, I was about to say, if nobody has any issues,
-->
let's hear the question.
-->
So it looks like if we had two different instances
-->
with different canvases,
-->
two people could be working on the same flow.
-->
If they're using the same registry.
-->
They can, they can.
-->
And when, if you were,
-->
if somebody is high speed enough,
-->
I don't know if this will work in this VM
-->
but you should be able to access my registry on,
-->
was it three, nine, yeah, nine.
-->
Ten dot zero dot three dot nine.
-->
I can't.
-->
Okay.
-->
Yeah, I was, I was wondering if these VMs would allow it.
-->
I don't, I don't think it would
-->
but if you wanted, you could add my registry
-->
and you could go back to your NaPhi canvas,
-->
create a new processor group, check that in
-->
and then while you're working on that,
-->
I can pull your latest version that you checked in.
-->
So I can't pull in real time,
-->
it's not like a Google doc or Office 365
-->
where we can work off of the same document
-->
but if you were to check something in
-->
and then continue working on it,
-->
I can pull what you checked in onto my canvas
-->
and I can continue working on it as well.
-->
So, you know, it does, you know,
-->
it doesn't provide that in-browser collaboration
-->
like you would see with a Word document
-->
but it does provide an ease of use to,
-->
for your teammates to, to all work on the same flow,
-->
for instance.
-->
Cool.
-->
My other, my last question.
-->
This may be out of scope at this training,
-->
I'm just curious how other people do this
-->
but if the flows are being backed up in source control,
-->
we're going to be running these as containers.
-->
Do we, do we need to back up
-->
any of the, anything other than the flow?
-->
Because I know there's like the data is queued
-->
and if there's errors and stuff.
-->
Do people typically back up stuff
-->
more than just the flow
-->
when they're running NaPhi in production?
-->
They do not.
-->
So that's, that's the, the beauty of this
-->
and the data flow, the queue, all of that,
-->
that's the, the actual data flow running.
-->
So when you, when you, you know, build your container,
-->
you will pull in your data flow.
-->
And so, and I can go into more of, of, of that.
-->
As a matter of fact, I'm writing it down now.
-->
Because in NaPhi, there is,
-->
there is a folder
-->
that you can use.
-->
So if you were deploying this
-->
and you were putting it into a container,
-->
you want to go ahead and kickstart the container
-->
with its flows.
-->
So you can actually, you know, pull the flow
-->
from, from GitHub or Azure DevOps
-->
and package that with your container
-->
and gun, you know, you just want to gunzip it.
-->
And so when NaPhi starts up, it looks for this file.
-->
And if it finds the flow.json.gz,
-->
it will extract that file and import the data flow.
-->
So now you've got a process set up that,
-->
you know, where you can, you know,
-->
you've got your flows, you know,
-->
managed appropriately during the version control
-->
and those types of things.
-->
And now you've got a process,
-->
you know, to build your container,
-->
that CSV process to build your container.
-->
You know, you will reference a data flow
-->
it needs to pull in.
-->
And so when that container is built
-->
and up and running,
-->
it's got the data flow
-->
that was in your versioning control.
-->
It's running with that information
-->
and you should be, you know, good to go.
-->
Minify, we're going to actually touch
-->
in some of this when we get to Minify
-->
because Minify does require you
-->
to bootstrap it with the flow
-->
that you want to use.
-->
There is no UI like I mentioned with Minify.
-->
So all of the flow building you will do
-->
for Minify is done within NaPhi.
-->
And so that's a great example
-->
where you may have a process set up
-->
where you're running Minify in a container.
-->
On a Raspberry Pi, for instance.
-->
And when you built that container,
-->
you need to feed it a flow
-->
that, you know, it will execute
-->
to ingest this log and format this log
-->
and send it over to NaPhi.
-->
You know, so that's how you would use that.
-->
But that's a great question.
-->
Did I answer it?
-->
Yeah, you did. Thank you.
-->
Okay, awesome.
-->
We have about five more minutes
-->
before, you know, let's break for lunch.
-->
Any other questions?
-->
Any, you know, even tips or tricks?
-->
You know, I like to, you know,
-->
do these classes where it's very casual
-->
and it's a very much conversation.
-->
You know, please pick my brain.
-->
You know, like I said, I've been
-->
through numerous huge NaPhi deployments.
-->
I'm one of the committers to this.
-->
So, you know, I know they are working.
-->
So if there's even an out of scope question,
-->
I'll be happy to answer.
-->
So we got about five minutes left.
-->
Any additional questions
-->
or we will just go to lunch a little early.
-->
Are we still planning to go over
-->
the multi-tenancy stuff?
-->
We are, we are, we are.
-->
Yeah, after lunch, I was going to have
-->
us work on one more flow
-->
and stuff like that.
-->
And tomorrow is going to be a lot of,
-->
you know, minify multi-tenancy
-->
and a couple other little things
-->
that we should need to cover.
-->
And that's also one of the things
-->
that you probably, you might have
-->
noticed is a Docker compose example
-->
in the, in your zip file.
-->
But we'll get into that.
-->
We do have Docker desktop
-->
deployed and those types of things.
-->
All right. Any other questions?
-->
Okay. If you wanted to try and like edit
-->
one of those, just to take a look at.
-->
Oh, great question.
-->
Great question.
-->
Let's go to download.
-->
And if you're looking at my screen,
-->
not five, one, two, six
-->
was released yesterday.
-->
I was wondering why I saw one,
-->
two, six earlier for, for,
-->
um, uh, for Brex, I think it was.
-->
And I was like, wait a minute,
-->
what's going on?
-->
So if you go to the not high download,
-->
you'll see the sources and the binaries.
-->
And so what we want to look at
-->
is the source.
-->
You can actually just download
-->
the whole source release as well.
-->
And that's going to give you
-->
the source code for not only the processors,
-->
but also the, um, not only the processors,
-->
but also NAFI engine
-->
and those other types of things.
-->
And then, um, you can actually look
-->
at Apache, you know, Apache links,
-->
their GitHub repository.
-->
You can see Minify was changed
-->
an hour ago, for instance.
-->
But if you look here,
-->
you will have all of the source code
-->
or all of the processors.
-->
One of the things that, uh, I mean,
-->
I would be happy you let me know,
-->
but is we can set up a,
-->
an environment to, to do, um, uh,
-->
you know, custom processor development.
-->
I was planning to potentially do that
-->
after lunch tomorrow after we get through
-->
Minify and some of the tendency questions.
-->
But yeah, you know, one of the things
-->
I like to do is, is, you know,
-->
quickly show you how to set up
-->
a dev environment for NAFI
-->
and how to develop your own custom processor.
-->
But if you're looking at the source code,
-->
it's available on GitHub.
-->
It's also available as a downloaded package.
-->
You, you know, you can see, um,
-->
if you do want to download
-->
and build it yourself, uh, the documentation
-->
actually goes into really good detail on that.
-->
Let's see here.
-->
Let's see here.
-->
Where's that documentation?
-->
So there's a whole developer guide, um,
-->
on how to, uh, you know,
-->
build a processor from scratch,
-->
um, and, and things like that.
-->
Um, it has a lot of information,
-->
uh, but you can go directly to GitHub
-->
download, uh, once you've got it downloaded,
-->
there's some things you can do to, uh,
-->
build it, uh, if you wanted to just,
-->
you know, mess around and change some things.
-->
If you want to build it yourself,
-->
building a custom distribution, uh,
-->
you can, you know, you can use Maven
-->
because Java's under the hood, um,
-->
and things like that.
-->
There is a MVM clean install,
-->
like include all, and that will include
-->
all of the, uh, NARS and all the bundles.
-->
You may want to, uh, only build, like,
-->
you know, certain bundles and things like that.
-->
There's actually another hidden, uh, option
-->
and it's the, um, rules engine.
-->
Uh, so there is a rules engine flag.
-->
They just don't list it here,
-->
uh, that you can also include.
-->
Um, but yeah, you, you,
-->
the developer's documentation and building it,
-->
um, you know, is really well documented.
-->
Um, and, and, you know,
-->
that leads me to another question.
-->
So, um, I've seen this, this quite a bit
-->
and I don't really go over it because,
-->
um, you know, um, to me, uh,
-->
you know, it's all in the install guide
-->
and things like that.
-->
But when you are installing this
-->
in and at, like, in a production type
-->
of environment, you want to make sure
-->
that you have, um, you know,
-->
all the hard file limits set,
-->
you know, the amount of files
-->
that you can have open on the, um,
-->
um, uh, system and those types of things.
-->
NAFA is, is really heavy
-->
on having a ton of files open.
-->
And so, you know, they will give you
-->
in the, um, instructions
-->
and I'm looking for it right now,
-->
they will give you in the instructions
-->
on how to set your, your hard limit,
-->
your soft limit on the open files,
-->
you know, those types of things.
-->
So let me see.
-->
But did that answer your question
-->
on where the source was?
-->
Yeah. Yeah.
-->
I think I might have more,
-->
but, uh, I'm moving away till we go
-->
over the process just about four to more, I guess.
-->
Um, is there anything I can answer
-->
real quickly now?
-->
Um, no, no, I think I,
-->
I can dig through that.
-->
I think I understand what I'm looking for.
-->
Thanks.
-->
Okay. Um, and, and like I said,
-->
I'll have more there.
-->
I can, I can, um, I can,
-->
I can type up some more, uh,
-->
tips and tricks on developing processors.
-->
But good question.
-->
Anybody else have any questions
-->
before we, uh, take a break?
-->
For me to eat early dinner,
-->
for you all, uh, get some lunch.
-->
Okay. Um, so let's, let's take a lunch break.
-->
It is two oh two.
-->
So let's just say it's two o'clock.
-->
Let's try to be back here.
-->
You know, let's take a, uh, a 42 minute lunch,
-->
43 minute lunch.
-->
Let's try to be back here by 12 to 45.
-->
You all your time to 45 central time.
-->
And, you know, we'll, we'll go into another,
-->
another day of flow and, and build that out
-->
and, uh, you know, get some,
-->
get some additional hands on and then,
-->
you know, call it a day.
-->
So have a great lunch.
-->
I will see you all in 43 minutes.
-->
Uh, and if you need anything,
-->
I'll most likely be at my desk
-->
or just chat me a message.
-->
Okay. Um, give me just a minute.
-->
Yeah, if you don't mind.
-->
Thank you very much.
-->
I don't know yet.
-->
Okay.
-->
Oh, do they?
-->
Yeah.
-->
I've just got an email.
-->
But it picks up a from 499.
-->
You can have your phone in for that.
-->
Uh, but the, the nonce coming out.
-->
All right.
-->
I'm just having to slow it down.
-->
Alright, I'm glad everyone had a great lunch.
-->
And hopefully everyone has made it back.
-->
Even given 43 minutes for lunch, I had barely enough time to finish.
-->
But we've got a good next scenario.
-->
Okay, so I have one minute after 2.46, my time.
-->
So let's get started. Our final activity for today is a scenario where you will be developing your own NAPA flow.
-->
I have put some tips, tricks, and some pointers in the scenario.
-->
But this is I don't usually like to give a test in any of the classes.
-->
This is what I like to see is just some hands on experience, you know, applying some of the things that we went over.
-->
So for this scenario, let me pull this up.
-->
So for this scenario, you will be.
-->
If you have the Dropbox link still or the the the ether pad pulled up, you can actually download the scenario information.
-->
Let me know if you cannot, but for this scenario.
-->
So for this scenario, you are a data analyst at a local government agency responsible for monitoring environmental conditions across various locations in the region.
-->
Your task is to aggregate, transform and analyze weather data collected from multiple local weather stations to provide daily summaries and alerts.
-->
So if you can, this scenario is is in the Dropbox link.
-->
It's also in the ether pad.
-->
Right here, so I can actually.
-->
I can go on to everyone's desktop and download it if you need help.
-->
But if you can.
-->
Try to grab this link if it's working.
-->
Nope.
-->
Oh, it's logging me in copy link address.
-->
OK, so if you can go back to the ether pad, I will also send this link in teams chat.
-->
There should be a zip folder that is.
-->
Not five scenario.
-->
It has the scenario as well as three data files.
-->
We are pulling from three different weather stations.
-->
It gives a report every hour.
-->
There is a data description in this scenario to describe the fields as well as, you know, like I mentioned some tips and tricks for that.
-->
So let me see if I can download this to my desktop.
-->
Go to your Dropbox.
-->
No, it's not.
-->
Oh, perfect.
-->
So if you take a look at that link that I put in, you should be able to download the scenario.
-->
Should be pretty easy.
-->
And then during that scenario, you just want to be able to go to your downloads.
-->
I am going to copy it to my desktop.
-->
You know, go ahead and extract the zip file.
-->
And.
-->
So, again, for this scenario, I've included the PDF of the scenario.
-->
So your objective is used to automate the collection, transformation and reporting of weather data from local files,
-->
simulating real time data ingestion and processing.
-->
I actually stood up a small web server to serve these files.
-->
But it requires a lot of SSL connections and everything else within nine five.
-->
So I decided to go with just a local file.
-->
But you should see this scenario as well as three data files, two CSVs and a JSON document.
-->
So if you can extract this information, go to your not by canvas, create a new processor group and.
-->
You know, start building your flow.
-->
What I'm looking for here is is just a recap of everything that we have learned with not five and registry.
-->
You know, because today is probably, you know, we'll touch on some stuff tomorrow.
-->
But tomorrow is mainly minify, you know, multi tendency, advanced, you know, deployment capabilities and also some some custom processor how to that will go through.
-->
So for this scenario, you're collecting the data, you know, the data is the CSVs and JSON.
-->
You're going to, you know, set up not five to monitor a directory ingest files as they appear.
-->
You know, those types of things I have put like a not five task.
-->
I didn't spell out how to do it.
-->
You can, you know, use the processors that you have available to build this flow.
-->
You can make it as complicated or as simple as you would like.
-->
Again, what I'm looking for here is a completed data flow that picks the data up, provide some enrichment, some aggregation.
-->
And then also, you know, that alert generation, you can just send the alert to a log message.
-->
You know, for instance, we do not have, you know, for fun or for me, you know, those types of things installed, you know, where you would typically see this in the real world.
-->
Well, that being said, feel free to get started.
-->
Does anyone have any issues getting the scenario?
-->
Good deal.
-->
So I'm here for questions.
-->
Feel free to ask any questions you may have, you know, any tips or tricks or anything that you'd like to see.
-->
There should be information within that PDF to help you out.
-->
The data structure is pretty self-explanatory, but if you need a breakdown of that, you know, I can send over something as well.
-->
And while everyone works on that, I am going to go mute, but I'm going to kind of pop into everyone's desktops, see how things are going and just provide any commentary as needed.
-->
Wish you were sitting on my face right now.
-->
So we have about two hours left for today.
-->
This may take a little bit of time.
-->
So what we'll do is spend the next hour on this flow development.
-->
Again, ask any questions along the way.
-->
We'll take a quick bio break or final break of the day and, you know, come back answering questions and review your flows.
-->
Again, you know, there's many ways to skin a cat with 9-5.
-->
So I'm really looking at, you know, some of the thought process behind, you know, developing your data flow and just that story that you have.
-->
I have a question. So for step three, it says execute script. Do we have a script for that?
-->
You do not. So if you do not have a script or, you know, you may not know scripting, feel free to use another processor to extract that, to deal with the, you know, that type of data.
-->
If you look through the processor, you know, the list of processor you have, you may want to be able to do like, you know, extract that as a CSV reader and a JSON reader and then put those back together.
-->
For those that knows how to execute like a script, you know, it's an option, you know, so do the best you can.
-->
For this one, you can, you can use custom scripting like there's also a jolt transform script.
-->
Python, I don't think we have Python installed here, so you may not be able to use a Python script.
-->
You might have to use a processor to just extract the data.
-->
And then once you have it extracted and saved as an attribute, you can then use those attributes to build your final output.
-->
And if you get hung up on that step or some of the other steps, you know, just, just ask me and we can, we can find a processor to do the function that you're looking to do most likely.
-->
Well, this scenario can definitely be accomplished, you know, utilizing all the processors that NaPy comes with out of the box.
-->
Not all the processors, but you wouldn't need any custom processors.
-->
Also, remember the documentation has a list of processors and their functions.
-->
You know, so if you are trying to find a processor to to simplify what you're trying to do or to do that function, feel free to reference, you know, documentation.
-->
A list of the processors and what it can do is there.
-->
I assume some of you will probably use the extract text processor as well.
-->
So, you know, that's there as well as, you know, some of those things.
-->
Sorry.
-->
Okay.
-->
Okay.
-->
So is this kind of like however we want to approach this?
-->
Yes, it's however you want to approach it.
-->
So like I said, yeah, you can learn from the previous example.
-->
And that's the reason we have both JSON and the CSV.
-->
So if you want to ingest CSV converted to JSON and then look up the JSON values together, I'm fine with that as well.
-->
Yeah, it's this again, you know, this is just to test the, you know, the skills that we've learned over the last day and a half and put those into practice.
-->
Do the best you can.
-->
You know, there's many ways to build a flow.
-->
There's many ways to do this.
-->
As you, you know, as this scenario proves that, you know, you may use, I bet when we pull this up, we're going to have folks use one set of processors.
-->
Others use another set of processors, you know, and it's all going to accomplish the same goal of getting that alert up.
-->
But yes, use any processor.
-->
You won't use previous processors.
-->
You've learned about those types of things.
-->
There's a reason I didn't make it all CSV is so you just can't reuse the last flow.
-->
But but yeah, you know, reuse what you need to.
-->
If you had Python set up properly and some of those things, you could easily do this in the script.
-->
If you knew Python, you know, so so yeah, you don't use the processors that's available.
-->
Do the best you can and use I put some trick tips in there.
-->
But if you want to use another processor habit.
-->
So just to see if I was on the right track with this, I was using an extract text and I created some properties based off of capture groups.
-->
Okay.
-->
I just call it call.
-->
So I have like called one through called 20 as attributes.
-->
Is that is that on the right track without extract text?
-->
It is.
-->
So, you know, one of the things that would put you on the right track is extracting these values as attributes.
-->
Because once you have them all as an attribute, you can you can manipulate all day long.
-->
And you can pass those attributes to different processors.
-->
You know, those types of things.
-->
But you are definitely on the right path.
-->
Thank you.
-->
Yeah, no worries. Good question.
-->
Hey, Josh, I got a question.
-->
Yes, sir.
-->
I have a feeling this is you went over this in the part that I missed this morning.
-->
But I'm trying to use the convert record processor and I'm getting a that it was validated against a different do it and the one I'm using.
-->
I'm not sure what the issue is.
-->
Oh, let's look at let's let me pull yours up right quick.
-->
And while I'm off mute, we still need to take our last break after lunch.
-->
But what I'm thinking is, is after I answer this question, I want to just go to the restroom and come right back.
-->
So, you know, just take a break in place and continue working on your data flow.
-->
Okay, record reader validated against this controller is disabled, right?
-->
Okay, so I'm not sure what that means.
-->
Yeah, no, you did.
-->
So go into the go into that processor and you see you have the CSV reader in the CSV are the Jason record set writer to the right is that arrow.
-->
There you go.
-->
And yes, you have those disabled.
-->
So you will need to enable with the lightning.
-->
Both the service, you can just do the service for now until you enable both of those.
-->
And I think you might miss this during the morning session as well as if you're trying to do like an Avro schema or something, you will need to put those services in there.
-->
If you want, you could if you have that zip file, well now see it's working now.
-->
But if you're using a schema, if you have that zip file that I sent that I had everyone download earlier, it actually has the work in flow.
-->
Okay, so it has that working flow in it.
-->
You know, go back and go to the.
-->
Oh, yeah, that's kind of what I've been.
-->
Okay, yeah, that's it.
-->
There you go.
-->
So it has that working flow.
-->
The reason that the CSV to JSON convert record is disabled there is only because, you know, you imported it in.
-->
But nobody ever enabled the controller service.
-->
If you go into that one, for instance, and go to the CSV reader and hit that, you see that you need to enable those and then your previous flow will work.
-->
And that should help take away the the warning.
-->
Yep, you're good.
-->
So the other two are disabled as well.
-->
Are you going to go back to your?
-->
Yep.
-->
The easiest way is just go into the processor, go over to your controller services and the bottom two are disabled as well.
-->
So you want to enable them and there we go.
-->
So that original data flow should work for you now.
-->
Okay.
-->
And you can tell actually the convert CSV to JSON.
-->
Just tell it to run once, right?
-->
Click until it to run once and see if it gets success.
-->
I don't think I did any of the steps.
-->
No, no, no.
-->
You already have a file in the queue.
-->
Oh, yeah.
-->
And then, yeah, hit refresh on your canvas.
-->
Oh, you do have success.
-->
Perfect.
-->
So you took that original CSV and you made it in JSON file.
-->
Cool.
-->
Okay.
-->
And you can use, you know, I don't I don't want to see like an exact copy of what we did this morning, but you can use this morning's activity as reference.
-->
Yeah, I missed the whole morning.
-->
Oh, no worries.
-->
No worries.
-->
Okay.
-->
Good.
-->
Any other questions?
-->
No, I think I'm good for now then.
-->
Okay.
-->
I am actually just going to run to the restroom.
-->
I'll be I will be right back.
-->
If I miss anybody, you know, while I'm away, you know, just leave me a message in the chat, but I'll be back in a couple minutes.
-->
I'm back.
-->
If anyone has any questions, let's spend about, you know, 20 more minutes on this.
-->
Hopefully make some progress and then start going over some of the data flows.
-->
If you get done, you know, with your data flow, let me know and we can start reviewing it.
-->
Looks like some of you are getting very close to completing this task.
-->
If you get hung up or, you know, if you're just taking a while, don't feel bad.
-->
We have plenty of time.
-->
You can work on this later, you know, after the class if you want or tomorrow morning.
-->
You can finish it on your own time later.
-->
I can send you the scenario.
-->
We'll be sending the scenario.
-->
So, you know, you can practice it on your own time as well.
-->
But we'll give it another 15 minutes and then let's start going through some of these data flows.
-->
Okay.
-->
Okay.
-->
Okay.
-->
Looks like everyone's very close.
-->
So we may just finish up working on this today.
-->
And then I can go through the flow later and see how you do.
-->
And then we can talk about it in the morning as well.
-->
So we have a few minutes left.
-->
I'm going to touch on some of these and see if there's anything I can do to help.
-->
The first one I have to pull up is Cody.
-->
How are things going?
-->
Anything I can do to help?
-->
Yeah, I tried to change.
-->
I tried to reuse the one we did in the previous flow.
-->
And I was trying to change some of the schema.
-->
But it was giving me an error on some of the formatting for that.
-->
Okay.
-->
So you're getting your weather data.
-->
Are you just getting the CSV files?
-->
Yeah, I'm just grabbing just the CSVs.
-->
Okay, perfect, perfect. Oh, awesome.
-->
And then you were setting the schema name?
-->
Yes.
-->
Do I need to add?
-->
Yes, just the weather.
-->
Okay, perfect.
-->
The last one we did was inventory.
-->
And then here, this is where it errors out.
-->
Let's stop it.
-->
I get an error.
-->
Temperature.
-->
The input string.
-->
Temperature.
-->
Yeah, click here.
-->
Okay.
-->
I noticed something already is weather capitalized on your other one.
-->
And here it's lowercase.
-->
Also, you can expand on that box.
-->
If you go down just a little bit below the down arrow on the box itself,
-->
to your right, right above.
-->
Okay, right there.
-->
There you go.
-->
That would have been helpful.
-->
Okay, so type record, name, weather, field.
-->
This one I had is an integer,
-->
but it was starting there and I changed the string and it passed.
-->
So I'm not sure.
-->
No worries.
-->
I mean, you can have them all at string if you want to.
-->
Okay, so I'm looking at the station ID,
-->
the date, the hour, temperature, humidity, wind speed, and precipitation.
-->
Okay.
-->
Say okay there.
-->
That actually looks good.
-->
Well, I did get an error on the caps mismatch.
-->
Say apply there and hit X and just close that window out.
-->
And let's look at your set schema, the step before.
-->
That's where I saw the capital.
-->
You got to stop and configure.
-->
There you go.
-->
Apply.
-->
Perfect.
-->
Okay.
-->
You need to enable your services again.
-->
And then the other ones are invalid,
-->
so let's look at why they are invalid.
-->
They may be invalid because of that service.
-->
There we go.
-->
All of them is enabled.
-->
Still throwing an error.
-->
Getting mixed for the temperature error.
-->
Let's go back into your convert CSV to JSON.
-->
Let's go to your CSV reader.
-->
Pull that controller service and let's look at the settings for that.
-->
Scroll down.
-->
You want to treat your first line as header.
-->
It's true.
-->
Because it doesn't know how to, there you go.
-->
So because it's trying to treat it as some of the data,
-->
so you want to set your first line,
-->
treat first line as header.
-->
Apply.
-->
Exit out of that.
-->
We'll enable it.
-->
Then next, what's our latest error?
-->
Did not parse.
-->
Failed to stream.
-->
Error while getting next to record for input stream 14.8.
-->
Not sure where that input stream would be.
-->
That is the temperature.
-->
So it pulled in the station ID, the date and hour,
-->
and then the temperature it was having a problem with.
-->
Change that to a stream maybe.
-->
Change that to a stream.
-->
We can just try to stream, not to get too over complicated here.
-->
Perfect.
-->
Put them all to stream.
-->
Make it a little easier.
-->
So when you go back to your canvas,
-->
that processor is going to be started because you started both the processor
-->
and the controller services.
-->
No, it's fine though.
-->
But go ahead and stop there.
-->
That's this.
-->
And then I haven't had him get to move.
-->
I just laid everything out.
-->
I haven't really built out any of the configurations for anything else.
-->
Okay, no worries.
-->
Like I said, I suspect some people will get done with this.
-->
Some people may not.
-->
But I'm glad you kind of laid it out.
-->
And so what we'll do first thing in the morning is kind of go through
-->
what your thought process is on some of the ones that you didn't get finished.
-->
And we can talk through those real quickly.
-->
Again, I'm not really looking for the flow to be completed.
-->
This is interactive.
-->
Just let me know what you're thinking and how you would do this.
-->
And just to make sure we have a good grasp on the components that we figured out.
-->
But besides that, it's looking good so far.
-->
If I was doing this real quickly, I would have used some of the previous flow as well.
-->
I would have saved that as JSON.
-->
I would have brought all three JSON documents in.
-->
I would have done an evaluate JSON path and extracted all the data from the JSON
-->
and then moved to the next processor to combine the data or make the calculations.
-->
So my flow would have probably been like six or seven processors if I was doing it a simple way.
-->
But I think you've got a grasp on this.
-->
Feel free to finish up.
-->
And then we'll go through it in the morning.