39:11
2024-05-06 08:57:50
3:11:28
2024-05-06 10:30:19
24:39
2024-05-07 07:20:20
19:49
2024-05-07 08:03:25
1:14:35
2024-05-07 08:35:13
15:41
2024-05-07 10:06:25
38:33
2024-05-07 10:37:03
2:48
2024-05-07 11:19:01
59:37
2024-05-07 11:33:56
6:10
2024-05-07 14:55:25
39:40
2024-05-07 15:02:44
9:45
2024-05-08 06:44:21
29:27
2024-05-08 08:09:24
2:51:40
2024-05-08 12:09:24
Visit the Apache Nifi - GROUP 1 course recordings page
WEBVTT
-->
was use cases and kind of going over some of those use cases those types of
-->
things I know minify is it can be a pain to work with and it comes in two
-->
different flavors so you have minify C++ and minify Java and so the
-->
primary differences and I'm bringing up some supporting slides for that
-->
but the the major differences in you know minify Java and minify C++ is
-->
basically minify Java is not my engine stripped of the UI and a couple of other
-->
components and but what that gives you is is what that really does is it
-->
gives you the capability to run most of your processors that you currently have
-->
or your custom developed processors on that machine so say I had a Raspberry
-->
Pi for instance and you know I had a bunch of custom processors I had that I
-->
have in my main instance I have you know some of those types things that's
-->
minify Java to deploy again you're going to deal with you know a lot of that Java
-->
JVM overhead as well as you know it's just not as fast as C++ and more
-->
low-level language so you know the minify C++ you can see you know that
-->
it's very very limited on the processors you get to use so you know
-->
luckily this list has expanded but if you notice there's not a lot of some
-->
of the other processors we don't have 300 processors here minify C++ is you
-->
know if you can get by with the processors listed here then you can
-->
you know utilize minify C++ all these processes are written in C++ as well
-->
you can write your own C++ processors I think there was a at least someone on
-->
the call that had C++ experience you know so when determining minify you
-->
know you know you need to take a look at what are some of your system
-->
what you know what kind of processors you want to use you know those types of
-->
things when determining you know which version of minify you want to use I
-->
would try to use not by C++ when possible you know just because of its
-->
lower overhead it is quicker it processes data better you know it's
-->
all around a better product but unfortunately you know they just added
-->
some Kafka support I think yeah it's in Kafka Kafka is used quite a bit in
-->
my circles and so you know we will use minify to consume or to produce
-->
data that goes onto a Kafka topic and those types of things again the
-->
version and we're about to actually hands-on walk through and installing
-->
some of this but the job version it can take almost every processor that you
-->
know your custom develop processors now if you develop a custom processor that
-->
has a dependency on the minify UI or some of those components that does not
-->
exist within the you know that would that doesn't exist in minify then you
-->
would not be able to use that processor it's rare that I run into a
-->
processor that has a total dependency on NAFA for example most of the time
-->
they're pretty agnostic and can run you know just about anything again
-->
you know it's small it's lightweight but it's also you know tedious to get
-->
going we're gonna try to get it going now and you know the configuration is
-->
not easy in those types of things so we are now going to bring up our
-->
desktop environment we should all have minify
-->
downloaded not that one all right so you should have a folder called minify
-->
one two five dot zero there's been let's go ahead and extract it
-->
yeah and you know it's pretty critical here we're not going to I don't want
-->
you to run it yet I don't want you to to kick anything off just because you
-->
know minify can let's kind of go through the the directory structure when
-->
you're installing and starting NAFA or minify you know you you have several
-->
options so let's look in the bin directory in the bin directory
-->
we do have application like minify and and bat files and those types of things
-->
Windows support is becoming more and more popular with minify I've seen
-->
minify used on government systems to act as the agent to do log aggregation
-->
WMI collection you know some of the cyber security aspects you know so you
-->
know if you if you see it in the wild you know you will potentially you know
-->
you'll see it there so in this you know you have a bat file to delete
-->
the service you can install a service I think I heard mentioned either
-->
originally or in the last couple days you're looking at potential like a raspberry
-->
pie you know that type of a device yeah I know there's seventh group for
-->
instance that's out of Tampa they're using Java minify on Windows laptops
-->
that are deployed a little outside the talk and you know in rerouting data
-->
back to the talk to a main non-fi instance and those types of things so you
-->
know depending on your use case you may want to you know use the Windows
-->
version I'd recommend you know if you can the Linux version just because
-->
it's easier to deal with it's widely supported you can use the C++ if you
-->
don't need custom processors and those types of things so it will then
-->
some of the directory structure you know we've got our been inside been you
-->
know we have we can do a dump of minify to see all the processes and in flows
-->
running that type of stuff we can do a delete you can delete the service
-->
after you installed it so if you have a you know if you have a service
-->
and you want to remove it you can you know do that we're you know we're not
-->
going to install it as a service you know for some of this hands-on but
-->
more of you know just looking at minify and getting it up and running those
-->
types of things minify C++ there's if you look around on the web you can
-->
Windows installer packages so you know you'll be able to install minify just
-->
like you would a regular Windows application which is really really cool
-->
so I know Cloudera supports that and and those types of things so if you
-->
need for that so install service you can define your your environment
-->
variables those types of things we can actually look at this
-->
one second
-->
we get a better editor than notepad
-->
Okay. So, anyway, so Webminify, you know, you can edit the bat files, you can manipulate
-->
those if you're working on a Windows environment, you can go in and set your environments, you
-->
can get a status, you know, just like you had with, and we touched on it only for
-->
a few minutes, but if you had the the actual nafi.bat file that ran the main nafi, it would,
-->
you know, it would, you can change username, you can change a few things there. Minify
-->
also has a toolkit, you know, so just like working with nafi, you can use a toolkit
-->
to work with minify as well. Okay, perfect. And if you look in the actual bat files,
-->
you can actually, you know, change where the PIDs are running, where your log directory is,
-->
those types of things. So, for you sysadmins out there, you know, you could set your home
-->
your job home, you can change all these and really get it customized to your location or
-->
your environment. The thing with minify again, there's no UI, there's no point and click,
-->
there's no ease of use when it comes to minify. So, you know, if you want to install
-->
this and get it up and running, please expect to hand jam a bunch of things in and, you know,
-->
deal with the hardest part with minify is actually getting it connected to a secure nafi. So,
-->
you know, getting the key store, trust store, those types of things set up. But if you're an
-->
admin, you know, definitely take a look at, in running this on Windows, definitely take a look
-->
at the bat files for this. The conf directory is really key with minify, you know, just like,
-->
you know, we could change, you know, nafi and do all kinds of things. You know, one of the
-->
things that we can do with minify is it has a configuration. We'll open that.
-->
And so, within this configuration, you know, it's, you have your flow controller name,
-->
any kind of comments, core properties, you know, file repository and how long it should be kept,
-->
you know, some of the swap characteristics. So, as you go through and start fine tuning
-->
some of these minify flows, you know, take that into consideration, you know, it's going
-->
to maintain a local repository, but, you know, even the default, you know, is 10 meg,
-->
for instance, and the max flow files is 100. You know, so you may, depending on how much
-->
data you are ingesting into nafi, you may have a need to, you know, you may have a need
-->
to expand upon that. If you do, you know, just take into consideration your content
-->
repository, your provenance repository, and those types of things. Your security, you know,
-->
some of your security properties will go in here, you know, your key store, your key store type,
-->
those types of things. You know, again, if you go through, in theory, even an invoked
-->
HTTP, you're going to need a key store and a trust store. A lot of times I see people just
-->
use the built-in key store and trust store that Java ships with. So, you know, I don't,
-->
if you change the password, which a lot of people forget about, you know, you should be
-->
decently secure to do that. But, you know, just kind of keep that in mind.
-->
Processor or processor groups, here's where you're going to feed those unique IDs,
-->
you know, for those, any kind of funnels and those types of things. There's a lot of
-->
configuration. One of the nice things, though, that they have released is the C2
-->
aspect of minifop. And so, let me see if I can pull up an architecture drawing of that,
-->
because I think it's pretty relevant here. All right. So, you know, what I'm saying is
-->
minify now has a C2 aspect that you can download. So, it's command and control.
-->
You know, minify runs a single data flow very well. And, you know, you may have a need
-->
for minify to communicate back to NAFA, you know, be able to get its new data flows and also,
-->
you know, get new configuration, those types of things. You know, so if you have a C2
-->
instance set up, that C2 instance is going to communicate back to NAFA. It can, you know,
-->
pull from the registry. And that's the reason that we went over registry. There's a lot of
-->
important aspects of that when you're talking minify and those services. So, it can actually
-->
pull that flow, push it out, update those flows, and manage it. Now, there's not a
-->
good UI to do this. And so, you're looking at, you know, some scripting. You're looking at,
-->
you know, potentially developing your own custom application and those types of things. So,
-->
just keep that in mind when you're using minify. You may grow a little frustrated at the,
-->
you know, the capabilities are there, but the configuration and easy use is not.
-->
The nice thing, though, is, you know, once you get it configured and up and running,
-->
you can make it as part of that CICD process. And if you're using Ansible or similar,
-->
you can deploy, you know, minify with ease. So, you know, when it comes to the architecture,
-->
one of the things that we usually see is minify, you know, I've seen two or three hundred
-->
minify instances reporting into one node, you know, reporting into a C2, for instance,
-->
and the C2 talking back to NAFA. You do not have to have C2. You can actually have directly
-->
from minify to NAFA, depending on, you know, how would you use C2 if you are,
-->
you know, changing data flows frequently, if you're trying to, you know, partition or assign
-->
nodes to different services and even capabilities. So, you know, kind of keep that in mind
-->
when you're building things out. You know, you may or may not need C2. Are you all running minify
-->
now? I know, Ben, you mentioned it, but I don't know if you were running it.
-->
Okay. Yeah, no worries, no worries. Okay, perfect. And then also,
-->
when we create a flow and put it into NAFA or minify, we are going to export that out,
-->
put it into NAFA, and I'll walk you through that. But, you know, that's how you would see
-->
a flow to minify. It's going to go into the configuration directory and, you know,
-->
run that designated flow. We will touch on that more as we go into it. Docs, you know,
-->
the sysadmin guide is there. You can, you know, kind of go through this as well,
-->
if you want to. There is a lot of information in this. I'm actually working off of the same
-->
one, but like an online version. You know, so, you know, if you want to save on space,
-->
if you don't need that, you know, just keep that in mind. There's not a lot of minify
-->
docker support yet. There is a couple of images I've seen that uses minify, but,
-->
you know, just keep that in mind as well that you may have to build your own docker container
-->
and write your docker compose. And if you move to a Kubernetes environment, you know,
-->
being able to control that. I do know that is, you know, there is minify in docker,
-->
minify in Kubernetes. I think the biggest problem I've seen across the board is just
-->
getting it to connect due to some of the hand jamming of security that needs to happen.
-->
Once you get that, you know, kind of fixed, you can again put that as part
-->
of your CICD process and, you know, it will definitely alleviate some of that hand jamming.
-->
So, some of the other capabilities within the configuration and those types of things is,
-->
you know, when many minify agents are running on the edge, you know, it's not possible to
-->
manually stop and edit, you know, the flow that you put in and then restart those.
-->
You know, there's a new component called the config change coordinator.
-->
And, you know, its ingestors were designed to automatically redeploy
-->
in response to a configuration update. And so, you know, if you have a C2,
-->
it can communicate out that configuration update or when minify checks in, you know,
-->
you will see that. And unfortunately, it's not something I can just point and click and show
-->
you. You know, that is, you know, it's a feature of minify that it does that.
-->
There's in the configuration, you know, how to do that. But, you know, that's, you know,
-->
how that's done. So, you know, keep that in mind. The config change ingester are the
-->
means by which an agent is notified of any potential new configuration. There's only currently
-->
three. That's like a file change ingester, rest change ingester, and a pull HTTP change ingester.
-->
You know, again, you're limited very much in some of the capabilities. So, if those,
-->
you know, requirements, you know, meet your use case, then please, you know, have at it.
-->
After a new configuration has been pulled and received, the ingesters use a
-->
differentiator in order to determine if the currently running config is different than the new
-->
config. You know, when the differentiator is used, it's configured for each ingester.
-->
There's only one differentiator in minify that you can use, and it's called the whole config
-->
differentiator. And all it does is compare the entire new config with the current running one.
-->
It does look byte per byte. And if a new config is determined to be new, the minify agent will
-->
attempt to automatically restart and pick up, you know, the configuration. So, for you,
-->
sysadmins, the bootstrap first saves the old config, you know, into a swap file. And so,
-->
the bootstrap monitors the agent as it restarts. If it fails to run properly when it restarts,
-->
it can revert back to an old config. So, you know, kind of, if you have a Tesla or,
-->
you know, I'm familiar with Android phones, you know, if an update doesn't succeed,
-->
it has a rollback capability. You know, I'm familiar with my wife's Tesla. It's the same
-->
with hers. You know, if an update doesn't succeed, it can roll those back, and that way,
-->
you know, at least your minify is up and running. So, you know, just for FYI.
-->
Data left in connections when the agent attempts to restart will either be mapped
-->
to a connection with the same ID as in the new config or orphaned and deleted. So,
-->
you know, kind of goes back to the question about the authorizers and those types of things.
-->
NAFA likes to use these unique IDs, these new UIDs. And when you're building a NAFA flow
-->
into a minify flow, and we'll go into some of this as well, you know, you've got to
-->
capture those group IDs. That's one of the bigger issues I see, is people will create a
-->
new flow, and when you do that, it's going to create a new UID for the connection. And so,
-->
any data that you were originally processing, if it doesn't finish and it's mid-flight,
-->
it's going to look for that new, it's going to look for that UID connection, and if it
-->
doesn't find it, it will just delete itself. So, you know, keep that in mind, you know,
-->
when you're building this out, mostly in you guys' type of environment. And I'm just imagining your
-->
environment based upon working with South Group last year and building out some capabilities that
-->
they wanted to build out and those types of things. The configuration for a one deploy is
-->
done in the bootstrap.conf, which you can also
-->
define here in the bootstrap configuration file. You know, the bootstrap.conf
-->
primarily revolve around the config change and gestures. The configuration of the bootstrap
-->
is done using the NaPy monitor notify and gestures key, which you will see
-->
a lot in this section. So, for your sysadmin, you can start seeing, you know, the
-->
file change notifier configuration I talked about, the rest change notifier configuration,
-->
HTTP, you know, and those types of things. This is where you would configure that.
-->
You can, since this is the Java version, you know, you can configure the JVM memory settings.
-->
It is very low out of the box, only 256 meg of memory, but you've got to also take
-->
into consideration that, you know, this is usually used on an edge device, you know,
-->
those types of things. You know, the C++ version is more efficient. If you're using a Raspberry Pi
-->
and Raspbian OS, the support for Minify++ is actually really, really good.
-->
Minify actually supports, you know, some of the Raspberry Pi capabilities.
-->
Let me pull that for you just so you can, because I know it was mentioned.
-->
Let me see if it's on this page. Yeah. So, if you're using the C++, there is built in some
-->
Raspberry Pi capabilities, you know, OpenCV is there, a USB camera, there's sensors. So,
-->
if you enable the sensors on the C++ version of Minify, you can read from the, you know,
-->
Raspberry Pi sensors. There's a lot of these that are built, the C++ versions of the processor,
-->
you know, is built for Raspberry Pi. It'll run on others, but it's, you know, specifically
-->
targeted after that. You know, you can use GPS. If you have a GPS module on your Raspberry
-->
Pi, you know, you have a GPS hat, it can pull that information, you know, those types of things.
-->
Capture, you know, PCAP, you know, things from, there's a, you know, I know of a lot of folks
-->
using this to capture PCAP data, you know, even on the edge, to filter and sort and
-->
analyze that network traffic. So, you know, just keep that in mind, you know, when you're
-->
using your version, that it's there, it's available in those types of things.
-->
Again, we're looking at the Java version, so you can, you have remote debugging. There is some,
-->
you know, for cluster mode to work properly, you know, some of the things that you need to do,
-->
you know, if you have some older, you know, system settings and things like that.
-->
This is, it's set to headless mode by default, you know, and those types of things. I was talking
-->
about some of the C2 capabilities. If you have a C2 server, you can go in and configure it.
-->
Again, if you are using a lot of edge devices where you're running Minify,
-->
I would highly recommend a C2 instance, just because once you get your Minify running with
-->
security, you know, you can feed that C2 the data flow that you want to run, and it will
-->
automatically download that data flow, run it, and, you know, and it'll just look for a new one
-->
periodically. So, you can set the heartbeat and all the information about C2 here. Again,
-->
I'm thinking about one of my senior engineers I was working with just a couple weeks ago. He
-->
was having a problem with C2 because there's not a lot of support for it online. You know,
-->
so a lot of the settings and configuration that you look at is, you know, trial and error,
-->
you know, you're just going to have to play around with it. But, you know, it does work,
-->
and you can get it up and running. Everything that we use to communicate to the C2, to
-->
you know, it's an SSL connection if need be and things like that. Now, when it comes to
-->
some of the fine grained security details of Minify, you're not going to see the same
-->
types of policies that Minify has. You know, just because it is a, you know, basically an
-->
agent of Minify, its primary purpose is to either, you know, get file, collect data,
-->
read a sensor, you know, you name it, do some quick operations. You may want to execute a,
-->
you know, a model, an AIML type model and those types of things. I do know,
-->
like Seventh Group uses it to capture images and run, you know, image classification models
-->
on edge and instead of sending up a, you know, 2, 3, 4-meg JPEG, they can just send the
-->
analytics from the, you know, the output of the analytics. You know, so that's something to
-->
keep in mind as well. Extensions is just like the extensions directory on NAFA.
-->
Actually, we did not go into that, but, you know, we talked about it I think on the first day.
-->
Your extensions is just, you know, just like you would in NAFA. This is where you're going
-->
to load, you know, your custom processor or something else like that. You don't want to
-->
restart your Minify agent. You just want to get it loaded and deployed. So, you know,
-->
use the extensions directory when you can. And, of course, there's the lib directory.
-->
If you see it, you know, we still have NARS, you know, listed here just like we would in
-->
NAFA. Not as many, you know, you would see within the NAFA, but, you know, you can,
-->
you know, depending on how you want to handle this, you can copy a NAR from,
-->
I'm going to go in and show you. So, you're able to copy the, like, one of the NARS from
-->
NAFA. Let's see, let's, there's already Kafka, there's solar. Like, we'll take the solar one,
-->
for instance. You know, if I had a Minify flow that was, you know, picking data up,
-->
sending that to solar. Solar is a search industry based off of Lucene,
-->
another Apache level project. I would include the solar NAR as part of my, you know, deployment.
-->
You know, so keep that in mind. You know, when you have your CSV process set up,
-->
if you do have custom processors, and it sounds like you may or you will,
-->
you know, you can have those built, and then when you build and deploy your Minify,
-->
you can use that processor on your Minify, you know, install. If you are, if you have that
-->
processor part of your flow file, and, you know, we have to design our flow file, you know,
-->
data flow itself, we have to design those within NAFA. And so once you, you know,
-->
get your data flow designed in NAFA, and you save, and we're about to do that right now,
-->
and once you save that flow, if that processor does not exist in Minify,
-->
it will reject that flow and not run. So, you know, just keep that in mind, you know,
-->
if you develop custom processors, or if you're using processors that are not part of the
-->
Minify distro. Again, you know, we know, you know, the C++ version, you have a very detailed
-->
list with Minify, and then also the, again, it's a very extensive list.
-->
So, if you, you know, these processors are not packaged with Minify, but it's able to use the
-->
following processors out of the box. You know, so execute SQL, we were using that, we were
-->
using, you know, some of these other processors in our data files, data flows that we built.
-->
So just keep that in mind. This list is available, but it's not readily available.
-->
And so what I'm going to do is put that as part of a presentation I send out afterwards,
-->
you know, just as a tip and trick that, you know, kind of reference this list based upon,
-->
you know, what your data flow may be. I think, you know, we would use a put file and a get file.
-->
You know, in our data flow, usually you would, like, you know, in a Minify instance,
-->
this is pretty new where they've, you know, kind of separated this out. There's a lot of,
-->
of, you know, there's a lot of thought in the future of Minify where you can
-->
just use NaPy and just deploy it as NaPy because it's actually the same engine underneath.
-->
So do expect some changes to this coming up soon. And that's because, you know,
-->
I just have the insider knowledge that these things are coming. So, but I'll send this list
-->
out. I'll send out, you know, a few things about it, but keep it in mind as well.
-->
Okay. So Minify is able to use the standard SSL contact service out of the box as well.
-->
If, you know, so keep that in mind. If you want to create a data flow with a processor
-->
not shipped with Minify, you know, the way you do that is set up your data flow,
-->
copy the NAR into the lib directory and then restart your Minify instance. And so, you know,
-->
just if you're at this, you know, you would have to, I think some of these
-->
NARs are packaged and bundled together. So if we look at NaPy, there is a NaPy.
-->
I'll have to go back and look and see where it's at. But there's a NaPy bundle NAR,
-->
and that is a bundle of some of these standard processors. And so, you know, with that being
-->
said, you may, you know, there is a JD bundle, but that's not it. So you may have to
-->
include the bundle of processors, even though you may not use the others.
-->
One of the ways around that is, is again, go back to the source code, get the specific
-->
processor. We looked at the Git file source code and, you know, go from there. You know,
-->
once you get the Git file, you can compile it. You can use it in your main NaPy instance
-->
and then turn around and ship that with Minify. As a matter of fact, as long as it's the same,
-->
you know, processor, what I would do is build a Git file processor that's specifically used
-->
for NaPy that you just extracted from the NAR bundle and, you know, use the Git file
-->
that's built into NaPy. So that way, you know, when you update NaPy, you can,
-->
you know, just update the bundle and everything else. And then you can specify the processors
-->
that get loaded on the Minify. Also, it would not include processors you're not going to use.
-->
Do remember every time you start NaPy, it's loading all of these processors up.
-->
It's not necessarily consuming a lot of resources, but any time you add a processor to the,
-->
to Minify, it's got to load it. It's got storage resources and other resources,
-->
you know, that's there. Okay.
-->
Within the Minify directory, you have the, you know, the actual Minify application.
-->
With that application, you can pass it like a dash H to see all of the help files, the
-->
you know, you can monitor your status. You can, you know, get health and bulletin
-->
and those types of things. Check the connection of links. You know, sometimes you may have a
-->
link, you know, not acting correctly, those types of things. You may have a remote process
-->
group. And so you can check those types of things. In my, there's not a lot of,
-->
like I said, there's some of this documentation online for Minify, but in my email I sent out to
-->
everyone, I'll include some of this. A lot of the documentation is not part of, you know,
-->
your standard NaPy distro where, you know, it's actually laid out really nice and things
-->
like that. They'll send you to a GitHub repo and you need to read the reading. So again,
-->
it goes back to what I was saying, is Minify is great. It works well. It can be a pain to set
-->
up. It can be a pain to manage, you know, initially, but once you get some over some of
-->
those hurdles, I think you'll really, you know, you can potentially really enjoy
-->
Minify as an agent because, you know, you have the ability to execute those data flows on the
-->
system. Okay. Some of the other things about Minify real quickly, you know, it runs on
-->
Linux. It runs Red Hat. Ubuntu is supported. Dimian is supported. Raspberry Pi OS is
-->
more and more supported and that's what we are going to use just to create a data flow and get
-->
it started. You know, it can be, again, a little tricky. So if everyone can, let's go ahead and
-->
launch your NaPy instance and bring it up and let's start creating a new, we're going to just
-->
I'm talking very simple. And so if we can, let's see here. I don't want to use
-->
it. So let's grab, oh, generate flow file. No, that's part of there. So bring up your
-->
process is to get this going. You know, it's probably going to error out just because,
-->
you know, your Minify will need, you know, if you create your own custom flow,
-->
it's going to need the processors, those types of things. But, you know, we can get started.
-->
Let's see here. Let's create a quick flow.
-->
So, okay, go to process group and let's say this is our Minify flow.
-->
Add it. Okay. Give everybody a second to get, you know, logged back in and back onto the,
-->
back into the canvas. Okay. So let's just use the log attribute. So what we're going to do is
-->
first we need a way to capture everything coming in from NaPy. So let's use like an all log
-->
attribute. And that's a little different than your log message. So what it is doing is,
-->
is we, you know, how this gets set up is Minify connects to NaPy. Minify, you know,
-->
Minify can connect to other external services as well. So, you know, if you, you do not
-->
necessarily need a, you know, a dedicated, I mean, you do need a dedicated NaPy instance,
-->
but you can have Minify deploy a flow that checks into your NaPy, but it will reach out
-->
and connect to another service running locally or say you have a service that's only available
-->
on the edge. And so you may have Minify reach out to that service, pull the data in,
-->
and then send it to NaPy. Okay. So, you know, the first thing we're going to do is have a
-->
log attribute processor. And then we are going to bring in an input port.
-->
So we are going to receive this from a remote site to site connection.
-->
This is what we get into with Minify. We'll output to a port and, you know,
-->
we are going to receive it. So name your port from Minify.
-->
And then when you are, I'm just naming mine, go into the create connection and settings.
-->
And, you know, now you have a connection on your site to site, you know, from Minify. So
-->
what would happen, and I doubt we get time to get all of this fully up and running,
-->
but what's going to happen is Minify needs a place to send, you know, its data,
-->
it needs a connection. So, you know, what we are doing is we are going to listen,
-->
you know, this is our input port. So we're going to listen for all those site to site
-->
connections from Minify. And when it comes in, we're going to log the attribute.
-->
You know, this is, you know, we don't have to log the attribute. We can bring whatever data
-->
is Minify sending us and, you know, send it to another data flow, for instance,
-->
send it to another processor group, for instance. There's a lot of capabilities here
-->
on what you're able to do, what you're able to configure, those types of things.
-->
So, you know, create your input port. Do a log attribute. If you want, you know, you can
-->
just do another log message.
-->
Add that to your canvas and add that success.
-->
Okay. And I'm going to auto terminate my log message so that way it clears.
-->
Okay. So everyone should have something similar. Perfect. Good, Pedro.
-->
Brett, you might have missed it. You want to have an input port dragged down
-->
and you want to do remote site connection site to site.
-->
There you go. Perfect.
-->
Give you just another minute to create this. But again, this flow is to receive data from
-->
Minify. So this flow is not going to go out. You know, this isn't a flow that Minify is
-->
going to run. This is our flow to catch whatever Minify sends out through its site
-->
to site capabilities and, you know, receive that file. I have a log attribute because,
-->
you know, it's going to, you know, Minify is going to send me attributes and data.
-->
So, you know, just have a log attribute, have a message, and it should be a valid flow.
-->
Okay. Good deal. Oh, some of you are getting fancy very quickly. Okay. So now that we,
-->
you know, that's our receiver, right? You know, that's receiving the Minify. That's
-->
what the data coming in. Now it's time for us to build on what we plan to deploy to
-->
Minify. So to do that, I want to go back and what I'm going to do is create a new
-->
processor group. And I'm going to call this my Minify deployed flow. You can name it
-->
wherever you want to name it. But, you know, just so I know the difference between
-->
the Minify flow I just created, you know, I can create that flow.
-->
Go into that processor group. Okay. And then once you're in, let's bring down our
-->
processors and we are going to generate flow files. Generate flow file. There we go.
-->
Generate flow file is used quite a bit. Pedro almost mentioned this earlier on
-->
your question about the web services. Generate flow file sometimes is used to
-->
kick off a flow. So, you know, if you have an invoked HTTP, generate flow file is
-->
nice where you can set a timer up, some other rules in the generate flow file,
-->
generate a message, it kicks off the invoked HTTP and that's what kind of drives the
-->
rest of that flow. But for Minify, let's just use the generate flow file
-->
processor. We are going to basically generate a zero byte file. I'm going to
-->
set it to text. And it's going to send hello world. Apply. Now, again,
-->
we are, this is the flow that will be deployed to Minify. So, you know, this
-->
flow is not to, you know, don't get it confused with making it valid within
-->
NAFA. It needs to be valid for the Minify install. So, I know that can get a
-->
little confusing, but, you know, just keep that in mind when you're developing
-->
your flow that, you know, depending on file locations and those types of things,
-->
you may use a git file or something similar. And, you know, that file
-->
location is not available because it's not, you know, configured on the, you
-->
know, the remote service. And then once we have that, we are going to now
-->
bring down, so we did an input port. So, let's do an output.
-->
Let's do site to site. And this is to NAFA.
-->
Drag your success over and go from there.
-->
The, let's see here. No, no. I apologize. Actually, delete that connection.
-->
We want to actually add a remote process group. And it's a, so that's
-->
the way when Minify sends it to the NAFA, it's sending it to that remote
-->
process group. You can, you want to put in your instance that you're running
-->
right now. So, HTTPS 127.0.0.1 colon 8443. And don't forget NAFA.
-->
So, when setting up NAFA, you know, we'll talk about this. You know,
-->
when setting up these connections, you can, you know, if you set this,
-->
you know, transport protocol to raw, you're going to need to work off of
-->
your own port. And if you look right here, because I get this question
-->
quite a bit. If you look at your NAFA
-->
configuration,
-->
in the NAFA configuration, you have, I think it's port 10,000 is the,
-->
um,
-->
there's a web port.
-->
The remote input port, remote input host. So, you know, when you are
-->
configuring your flow, you can choose raw or HTTP. If you use HTTP,
-->
which is the one I recommend, HTTP is going to communicate with your NAFA
-->
over the URL that you're logged into. So, it's going to create that secure
-->
connection back to your NAFA instance. And that's how it's going to listen
-->
and not identify as going to push its data. Now, a lot of people that are
-->
processing lots of data on the edge, you may not want all of that data
-->
going to the main NAFA HTTP port. The reason being is, is that
-->
Jetty server running underneath, it's serving up the webpage and the
-->
API and everything else can get overloaded. And so, it could cause
-->
your browser to become a little unresponsive. And so, you might not
-->
be, you know, you might turn a flow on, say 100 edge nodes, you turn
-->
that flow on. And, you know, once you turn it on, it's going to start
-->
bogging down your system. It's going to, you know, you can get
-->
into your UI, those types of things. I've seen that happen. The best
-->
way to mitigate that is just, you know, managing your resources. You
-->
know, but if you want to send it to a raw, you know, NAFA port,
-->
that is okay as well. The beauty of that is, you know, you can
-->
configure that port. You can, you know, do some load balancing.
-->
You put Nginx in front of this and, you know, do some really
-->
advanced load balancing and those types of things. I recommend the
-->
HTTP port though. It's just a little bit easier to deal with. You
-->
know, it operates on the same port in IP that you have in NAFA running.
-->
So, if you can imagine, you don't have to open another, you know,
-->
another hole in your firewall to allow access. You know, so keep
-->
that in mind. I always recommend HTTP. You can put some proxy
-->
information in, those types of things. But, you know, that's usually
-->
what I like to do.
-->
And then take your generate flow file, you know,
-->
connect to your port.
-->
It does not have any knowledge of inputs.
-->
Let me see what's going on with my other one.
-->
Hmm.
-->
Did you all receive the same error that it didn't have an input port?
-->
Yes, I got it. Okay. Thank you.
-->
Oh, I think it's because it's trying to connect.
-->
But again, I haven't set up the trust store, the key store,
-->
and all of that, you know, on minify,
-->
which is, you know, a little outside the scope of
-->
configuring NAFA. So anyway, so what you would do
-->
is set up your flow file and get that configured. And you're going
-->
to have this go to an output, that remote process group.
-->
Once that remote process group is there, you want to make sure you include
-->
your, you know, the URL, the HTTP. Do remember, you've got a UUID here.
-->
So if you need to, you know, do those types of things,
-->
you may have to define that in your CI CD process, you know, some of
-->
those types of things. So, you know, just keep that in mind.
-->
Um, let's see here.
-->
But actually, it should still connect.
-->
See if I can solve this, let's see here. I do have an input.
-->
Hmm.
-->
We have now two of them, so it should.
-->
It doesn't match any of them.
-->
I think we'll connect to due to SSL.
-->
Um, so anyways, you know, the issue I'm having here is the,
-->
looks like the SSL, you know, it's not allowing it to connect to the
-->
self because of SSL. It can't match the names.
-->
But when setting this up, you're going to create your flow file here.
-->
And then once you have your flow file, you know, define that we are
-->
going to export that out and run it in Minifile.
-->
So let me see if I can.
-->
This just worked earlier yesterday when I was testing it.
-->
One second, let's see. What am I blowing up? Put port in here.
-->
Yeah, but that's not going to do it because
-->
there's no definition of where it needs to go.
-->
I'll have to take a look and see why the
-->
configuration is blocking me when it was just working
-->
last night. But basically, you're going to,
-->
this is where you're going to build your Minifile flow.
-->
You're going to, you know, this is where you do your operation. So you,
-->
you know, right now we're just doing a generate flow file.
-->
And then we're just generating a hello world
-->
and sending it to the remote Minifile group.
-->
And so, you know, you would configure that here.
-->
So anyway, so I'll diagnose this and see what's going on here.
-->
But I think you get the principle behind it.
-->
So if we were on Minifile itself in deploying this to Minifile,
-->
we would take and do a generate flow file. We can chain together
-->
other processors, you know, say we were doing an Invoke HTTP
-->
because we are going to grab a local, you know, a local file
-->
that's running on the web server. You can build out your flow as much
-->
as you want or as little as you want. But at the end of that flow file,
-->
you're going to send that to a, to a remote process group.
-->
And so, you know, that remote process group gets defined here.
-->
You know, you want to make sure you have your,
-->
you know, your property set correctly.
-->
Good, lower latency.
-->
Is anyone else's latency getting bad on this?
-->
There we go.
-->
Oh, there we go.
-->
You know, just, just, you know, to reiterate,
-->
when you're configuring a remote process group,
-->
you're going to need to put, you know, your,
-->
you know, your NaFi instance that you're going to communicate with.
-->
You're going to need to set up the, you know, the matching certificates.
-->
So the certificates that you use for NaFi
-->
is going to have to be installed for your Minifile install as well.
-->
You know, for that, for this to work properly,
-->
it needs to make that, you know, that connection.
-->
Again, that's where you go into the Minifile key store or trust store
-->
and set those appropriately.
-->
I don't know why I configured this like yesterday
-->
to double check that it worked, and for some reason it didn't.
-->
But we get the idea.
-->
So once you create your Minifile flow,
-->
you can leave the group and,
-->
and then what we need to do now is actually,
-->
you know, get that flow.
-->
So we've, you know, created our data flow.
-->
Now we need to export this out.
-->
So if we say,
-->
if you right click on that and say,
-->
download flow definition with external services,
-->
you should be presented with a JSON document.
-->
And I'll bring it up so you can see what it looks like.
-->
Okay.
-->
Again, right, it's messy.
-->
So once you download your flow
-->
with those external services,
-->
this is the flow that you are going to use
-->
to feed your Minifile instance,
-->
that initial, you know, that initial flow.
-->
If you have previously set up a,
-->
if you have previously set up a C2 instance,
-->
you would actually go into Minifile
-->
and I configure Minifile to communicate
-->
with the C2 instance and the properties.
-->
And then you can pass the,
-->
you know, you can actually pass Minifile command
-->
that tells it what initial data flow to get
-->
because C2 is going to serve that up.
-->
But for this scenario,
-->
we're using Minifile to,
-->
you know, we're using Minifile to create our flow
-->
like you would do no matter what.
-->
While we can save that flow,
-->
you know, for this way of running Minifile,
-->
we can export that flow as JSON.
-->
Make sure you don't export it as like a CSV
-->
or not a CSV, but an XML template
-->
or any of those other formats.
-->
It is expecting a JSON document.
-->
And so once we have that running,
-->
we need to,
-->
so once you have that, we can actually copy that file,
-->
go into Minifile,
-->
into your conf directory and paste that flow.
-->
Now, even with this flow, it's not going to,
-->
it's not going to pick it up.
-->
Minifile is looking for a specific name for this flow.
-->
So once you have developed your flow in Minifile,
-->
you've exported and saved that flow,
-->
you know, can copy it into your conf directory
-->
and then you need to rename that to flow.json.raw.
-->
And so that is your initial flow of,
-->
of, you know, for Minifile.
-->
Again, you know, in the conf directory,
-->
we also have, you know, the config,
-->
you know, logging mechanisms,
-->
you know, your bootstrap.
-->
When you set up Minifile,
-->
you're going to need to install,
-->
no matter what, you're going to have to configure
-->
the security settings.
-->
You know, so it will not,
-->
Minifile will not connect to a secure NaFi instance
-->
if the security settings are not set up.
-->
And so I think when I made some changes,
-->
I might have changed some of my security settings
-->
and, you know, it just won't connect.
-->
I was hoping it would connect for demo purposes,
-->
but this is how you build a flow.
-->
This is how you export the flow.
-->
This is where you put it.
-->
You need to rename it to that flow.json.raw.
-->
And so, you know, that way, NaFi will recognize it.
-->
On a side note, I think I mentioned,
-->
NaFi will only use one flow per Minifile instance.
-->
You can, you know, you can build into that flow
-->
multiple operations, but what I can't do
-->
is I can't export the CSV to JSON demo flow
-->
that one and that one
-->
and expect to run all three of them in Minifile.
-->
Minifile runs one.
-->
So when you're building out your Minifile flow,
-->
take that into consideration.
-->
A lot of times we like to, you know,
-->
bust this up a little bit
-->
and do those types of things.
-->
In NaFi, that's great,
-->
but within Minifile, it's exactly one flow,
-->
one data flow.
-->
So, you know, make sure you export that out.
-->
If you, I wouldn't worry about putting,
-->
you know, any kind of beautification on this flow.
-->
If you do, just remember and keep in mind that
-->
whatever, you know, this Minifile deployed flow
-->
is going to exist on your NaFi canvas.
-->
If that's the process you have set up
-->
to deploy these flows, again, you know,
-->
C2 can take advantage of registry
-->
and some of the other services,
-->
but from a standard Minifile to NaFi connection,
-->
you're going to just build that flow
-->
in a new processor group.
-->
That processor group is just going to sit there,
-->
you know, so you will be able to,
-->
you know, you can do some of the beautification,
-->
you can do the labeling and the naming and all that,
-->
but taking into consideration that that flow
-->
is not being run within that NaFi instance.
-->
It's only being ran on the Minifile instance.
-->
Oh, that was a lot of information real quickly.
-->
So I'm going to pause there and, you know,
-->
give me what questions you have about Minifile
-->
that I can answer.
-->
Hopefully I didn't put everybody to sleep.
-->
Okay. So, you know, Ben, when we originally chatted,
-->
you also wanted to potentially hear
-->
some of the use cases for this and, you know,
-->
for these things.
-->
So I've written up some use cases.
-->
I can actually go into a real-world example for you.
-->
So I sit on the board of a company
-->
I founded a while back.
-->
I resigned myself for a CTO,
-->
and now I just sit on the board.
-->
And I was supposed to have a board meeting yesterday,
-->
but it got canceled.
-->
But I work one or two hours a month for this company.
-->
But my original design is,
-->
is what we do is we detect FOD on runways.
-->
So foreign object in debris on the runways
-->
for the F-35 program for the Navy.
-->
And, you know, so as you can imagine,
-->
we're reading radar data.
-->
We're reading, you know, other sensors
-->
and those types of things.
-->
And we're pulling that data in, scoring it.
-->
We've got numerous, you know, AI capabilities deployed.
-->
And so, you know, when we originally built
-->
this software to do this,
-->
I used the Minifile C++ under the hood,
-->
you know, to do some of that orchestration of services,
-->
you know, receiving the, you know, the radar data,
-->
pulling it in, executing a, you know,
-->
executing models and executing processes on that data,
-->
and then sending the alerts back to a centralized location.
-->
Radar data is about 25 gigs per every two seconds.
-->
So as you can imagine, that is, you know,
-->
there's no way you're going to stream
-->
that information to the cloud.
-->
Just not possible, you know, with, I mean, it's possible,
-->
but, you know, they don't have the budget
-->
for what it would take.
-->
So we deployed edge nodes to ingest that,
-->
utilizing NVIDIA GPUs and those types of things
-->
to ingest that data, bring it in,
-->
look for debris on the runway.
-->
And if it found something, we could locally,
-->
you know, send out a message, you know,
-->
right now they usually have airmen and sailors
-->
walking the runway.
-->
And, you know, humans have a tendency
-->
to make mistakes where machines are,
-->
you know, usually a little bit more accurate.
-->
And so they were noticing that, you know,
-->
debris sometimes would get left behind.
-->
And if you're familiar with the F-35 program,
-->
it has one engine.
-->
And so if that engine goes, then, you know, by plane.
-->
And so, you know, that's why they started using that.
-->
So that's, you know, a real world,
-->
you know, use case and strategy.
-->
Another one that I'd like to talk about
-->
is Seventh Group.
-->
We deployed some small compute, think Raspberry Pi,
-->
think Arduino, you know, Minify was rewritten
-->
to work on a low-level ESP32-type microcontroller.
-->
You know, so, you know, it would capture images,
-->
bring those images in, score those,
-->
image classification, facial recognition,
-->
those types of things.
-->
It would also collect data from other sensors,
-->
like, you know, your standard RTL-SDR-type hardware,
-->
which is your signals, you know,
-->
looking for time pressure monitoring systems,
-->
looking for Wi-Fi, you know, those types of things.
-->
You know, being able to bring that in,
-->
score that, and look for anomalies or patterns
-->
or, you know, just new devices that have popped up
-->
and, you know, securely send that to a NiFi instance.
-->
Some other use cases for Minify,
-->
you know, it's heavily used in the IoT industry.
-->
You know, Minify can be deployed directly on IoT devices.
-->
There's OPC UA is a protocol that, you know,
-->
SCADA, some of those low-level hardware devices use.
-->
So, you know, just a few years ago, for instance,
-->
I was on a Halliburton oil rig.
-->
I am in Texas, so, you know,
-->
we have lots of oil rigs around us.
-->
But I was on an oil rig, you know, setting up,
-->
you know, Minify that was part of, you know,
-->
a product to collect that, you know,
-->
some of that OPC UA protocol sensor data
-->
and doing the filtering and sorting
-->
and sending that back to NiFi.
-->
So that's another real-world example.
-->
I've got some use cases on Minify being deployed
-->
where the data needs to be collected
-->
from remote or difficult-to-reach locations.
-->
I already mentioned oil rigs.
-->
Weather station or Space Force Space Weather
-->
utilizes some of this as well in their environment,
-->
you know, to collect data from weather stations
-->
and others, you know,
-->
to just like we were doing in that scenario.
-->
Actually, when I was building that scenario
-->
is because I had, you know,
-->
some of the Space Force work
-->
I've done with them recently in mind.
-->
So that's another use case.
-->
You know, one of the others I like to deal with
-->
is, you know, in distributed systems,
-->
monitoring various components
-->
can generate a significant amount of data.
-->
Minify can process and filter this data locally
-->
and only transmitting alerts
-->
or summarized information to a central system.
-->
You know, a good example of this use case,
-->
it's going to sound like I'm crazy.
-->
I have around my house, I have, you know,
-->
cameras and those types of things.
-->
I do live on my farm,
-->
so I don't have a lot of bandwidth
-->
and I'm at the mercy of Starlink a lot of times.
-->
And some of these devices
-->
are connected via Wi-Fi back to my house.
-->
There's not a lot of bandwidth available
-->
what I would do is capture like license plates
-->
as they drive by, I do, you know,
-->
license plate recognition on it
-->
and that way I get just the text
-->
of the license plate in the state
-->
and to make a model of the car.
-->
And so anybody who drives down my road,
-->
I can capture that information
-->
and as you can imagine,
-->
a JPEG image can be up to about 10 meg.
-->
But if you look at the actual data inside that,
-->
you know, it's less than one kilobyte.
-->
So, you know, saving on data transmission rates
-->
and those types of things is another great use case.
-->
And then, you know, finally,
-->
Minify can help manage bandwidth usage
-->
by ensuring that only necessary data
-->
sent over the network, you know,
-->
we're seeing a lot more of that
-->
in the cybersecurity operations.
-->
I just helped the Navy on another project
-->
for AI operations where, you know,
-->
we are bringing in data from different sensors
-->
and those types of things,
-->
but we're making the decisions,
-->
we're pushing the decision making
-->
more and more out to the edge to bring that in.
-->
So we use Minify for that data collection
-->
to execute models and those types of things.
-->
Minify does have some like TensorFlow capabilities.
-->
Some of the previous versions of Minify
-->
had, you know, TensorFlow built in.
-->
If you are a, you know, data scientist,
-->
which, you know, Randy, you know,
-->
you and some others,
-->
I'm sure you know about the TensorFlow framework.
-->
And so, you know, being able to execute models
-->
while, you know, I have a TensorFlow processor,
-->
for example, and so when it takes an image,
-->
it will send it through the TensorFlow processor
-->
and make decisions and send just the decisions out.
-->
You know, so streamlining, you know,
-->
some of those data flows is another use case.
-->
I will email these five or so use cases to you all.
-->
I know that was one of the questions I got early on.
-->
But with that being said,
-->
I know we had a hiccup here with Minify,
-->
not wanting to come back to the instance.
-->
But any questions on Minify use cases
-->
or those types of things?
-->
So I'm not sure if you mentioned this.
-->
From my understanding, Minify is just for like edge devices.
-->
Do they, like, could we deploy it
-->
in the data center as part of the cluster?
-->
Absolutely.
-->
Because I'm assuming they would take up
-->
less resources than full nodes, like full MAPI nodes.
-->
You nailed it, Brett.
-->
So I, you know, like I think I did mention is,
-->
I do see a lot of organizations installing,
-->
you know, if you probably heard of Splunk,
-->
you've probably heard of some of these other tools.
-->
They all have agents.
-->
They want to install on Windows.
-->
But, you know, what we are seeing
-->
is more organizations going to Minify
-->
and installing that as a service
-->
as part of the Windows laptops.
-->
And, you know, collecting, you know,
-->
WMI information, you know, other information.
-->
Potentially executing other agents
-->
and those types of things.
-->
So, you know, Minify was built for the edge,
-->
but also Minify is extremely small
-->
and extremely efficient, especially the C++.
-->
So if you have a use case where you need an agent
-->
on a server to do data orchestration,
-->
that is a perfect use case for it.
-->
So when you hear Minify,
-->
don't just think Raspberry Pi,
-->
you can do this over, you know,
-->
hundreds and hundreds of servers.
-->
Can it be joined to the same cluster
-->
with a Zookeeper and all that
-->
as a regular Minify node?
-->
So it's going to connect back to Minify.
-->
So because Minify is not distributed and scalable,
-->
like a NaPy instance,
-->
you know, think of Minify more as your agent
-->
that you install.
-->
It doesn't use Zookeeper
-->
or any of those types of tools
-->
just because, you know,
-->
it's more of an agent than a distributed application.
-->
Now, with that being said,
-->
if you had, you know,
-->
say you had 10,000 servers
-->
and Minify C++ was your agent
-->
and, you know, you had them all running,
-->
you know, a subset of flows,
-->
you know, your NaPy instance,
-->
which you've now, you know,
-->
went over some of the scalability,
-->
multi-tenancy, some of those types of things,
-->
you know, depending on the amount of data coming in,
-->
can, you know, distribute that workload
-->
where, you know, under the hood,
-->
you may have, if you have 1,000 servers,
-->
you may have, you know,
-->
100 going to one NaPy instance,
-->
you know, another 100 going to another NaPy instance,
-->
so forth and so on.
-->
So, but yeah, no,
-->
it does not have Zookeeper,
-->
you know, or distributed capabilities like that,
-->
you know, so you would just use it as an agent.
-->
Great question, though.
-->
Go ahead.
-->
Thanks.
-->
Yep.
-->
No, you just confirmed what I thought.
-->
Yeah, yeah, yeah.
-->
No, and that's, you know,
-->
when people ask about Minify,
-->
they always think Raspberry Pi,
-->
and I'm always like, no,
-->
we can use Minify on a huge server.
-->
You know, there may be a piece of data
-->
or a service you want to call
-->
that's outside of your typical processing,
-->
and Minify makes a great use case for that.
-->
Okay, all right.
-->
What other questions for Minify do I have?
-->
And I'm kind of hoping for quite a bit.
-->
Okay, well, if I don't have a lot of questions on Minify,
-->
what I'll do, like I said,
-->
I have documentation on this.
-->
There's documentation available online.
-->
I have some other documentation.
-->
I will include that in my presentation,
-->
and when I send that presentation out,
-->
that way you can have a reference.
-->
You know, if it had worked with the built-day flow,
-->
but the only difference you would have seen there
-->
was receiving data into Minify.
-->
But it's the same concept that we have with the flow.
-->
So, you know, once...
-->
Yeah, even like this flow, right?
-->
You know, it would have been input port and coming in,
-->
and then it would go through your traditional data flow.
-->
The Minify and NiFi,
-->
the biggest thing that they have in common
-->
is just that ability to use a site-to-site connection.
-->
If you are setting up Minify, you know, take...
-->
If it takes a while,
-->
if you keep running into certificate issues
-->
and things like that,
-->
do know it's kind of common and typical.
-->
The LOE to set up Minify the first few times
-->
is pretty high, but, you know,
-->
once you get it where you need it
-->
and you get it configured,
-->
and, you know, as part of some of your other processes,
-->
it becomes a lot easier.
-->
If you get C2 up and running,
-->
and those capabilities, that's nice as well.
-->
Like I said, I was just working on one
-->
a couple of weeks ago,
-->
and it was a pain to get running
-->
just because of some of the security settings
-->
and things like that.
-->
They don't make this easy when it comes to C2,
-->
Minify, those types of things.
-->
And it's definitely on the roadmap to make it easier,
-->
but it's pretty difficult, you know, as it is.
-->
The next class is the, you know,
-->
as my understanding,
-->
is not necessarily the more
-->
technically advanced class,
-->
so we won't even probably touch on,
-->
you know, some of the core technical challenges,
-->
you know, that these things have.
-->
But with that said, any other questions?
-->
Hey, I don't know if I missed it.
-->
Did you mention that Minify
-->
can't connect to registry as well?
-->
Because when we did registry stuff,
-->
it was all gooey on the canvas.
-->
Yes, yes.
-->
So, no, I think I mentioned it at the beginning.
-->
Minify can connect to registry,
-->
and C2 can connect to registry.
-->
And that is another reason
-->
that if you're not using registry currently,
-->
I don't know if you are not,
-->
I would highly recommend it,
-->
because once you get registry installed and in place,
-->
it's going to be one of those foundational items
-->
you will need to finish deploying
-->
the rest of the architecture.
-->
Do you use registry?
-->
We have it as part of our deployment,
-->
but it's, I don't know if we're going to do it
-->
in production right now,
-->
because it's in anonymous mode,
-->
and it's HTTP.
-->
Just like the one we installed, right?
-->
Yep.
-->
Yeah, we got the nodes secured,
-->
but we haven't secured the registry,
-->
so it may not be in iteration one,
-->
but we're planning to deploy it.
-->
Oh, okay, perfect.
-->
Yeah, I mean,
-->
Minify, like I said,
-->
Minify always comes secured now
-->
just because of the issues with it,
-->
but registry out of the box is not secure.
-->
And once it does become secure,
-->
you make registry secure,
-->
you're going to have to go into Minify
-->
and give it Trustor, KeyStore,
-->
those types of things as well.
-->
So kind of keep that in mind
-->
when you start updating the security on registry.
-->
But it's also another, like I said,
-->
another good reason to use registry
-->
is just because of C2, Minify,
-->
those types of things, right?
-->
Great question, though.
-->
Great question.
-->
So how do you connect to the registry
-->
on Minify?
-->
Like how would you import a flow
-->
from a registry on Minify?
-->
Do you still just do it in the canvas
-->
like we were doing
-->
and then just export?
-->
So what you would do is,
-->
of course, you've got to start with,
-->
you know, you've got to start with
-->
NAFA to build the actual flow, right?
-->
And then once you get the flow built
-->
and installed into NAFA,
-->
you can, what I've seen is
-->
the Minify C2 will connect to registry.
-->
And so when you specify, like,
-->
so a C2 server can use the registry settings
-->
and that way when you deploy your Minify,
-->
it can look at C2 and, you know,
-->
you may bootstrap it.
-->
Here's the name of the flow
-->
that you want to use from registry
-->
and it will pull that flow down through C2.
-->
C2 can feed it to the Minify agent
-->
and Minify will run it.
-->
Any other questions?
-->
I'm going to write down that question, though,
-->
about Minify C2 and registry.
-->
I may have an image on that,
-->
on how that works.
-->
I'm going to take a look.
-->
We're going to take our last and final break
-->
before we come back
-->
and talk about developing custom processors
-->
and then, you know, wrap up with,
-->
you know, just some next steps
-->
and things like that.
-->
And then, you know,
-->
save a few minutes for questions.
-->
If we get done a few minutes early,
-->
we can, you know,
-->
get back to your day job if you want.
-->
But, you know, we'll take a break.
-->
I will see everyone back here in 15,
-->
which is 2 o'clock, your time.
-->
And then we will go over
-->
custom processor development
-->
and Q&A.
-->
It wasn't weird you got it on only, like, one hour.
-->
He is an idiot.
-->
Hey, you know, hopefully everyone's getting back from our last break.
-->
We'll spend a few more minutes
-->
just going over some tips and tricks
-->
and as well as, you know,
-->
some custom processor development.
-->
I do have a question.
-->
Ben, if you've made it back,
-->
or anybody from the audience,
-->
would you have felt better
-->
if you were working on your own laptop
-->
versus this virtual environment?
-->
And do you all have that capability
-->
to work on your own environment?
-->
The reason I ask is this.
-->
Maria shot me a note, you know,
-->
because she noticed, you know,
-->
some of the latency issues we were having,
-->
some of the technical issues
-->
we were having with the virtual environment.
-->
So I don't know if everybody uses,
-->
like, a government laptop
-->
and they can install software or not,
-->
or your own personal laptop.
-->
So, Ben or someone else, you know,
-->
what would your suggestion be?
-->
Oh, okay.
-->
I mean, but I think anyway,
-->
I mean, are you all able to work on your own laptops
-->
or desktops or, you know,
-->
if we were to have this class
-->
without the DAW desktop environment,
-->
would you, you know,
-->
could you still support installing
-->
NaFi and others on your personal,
-->
on your laptop that you're working off of?
-->
Can I ask the question again?
-->
Some of us are appointed essays,
-->
but not everybody in this,
-->
taking this course, I think, would be able to.
-->
Okay.
-->
Specifically not next week.
-->
Yeah.
-->
Just the following one.
-->
I don't think everyone there will be an essay on their system.
-->
So they'll probably run into quite a few issues.
-->
Okay, okay.
-->
So that, because that was the question,
-->
is, is, you know, you know,
-->
can you, because, you know,
-->
you guys are on the internet,
-->
some, some, you have a lot of safeguards in place.
-->
Yeah, if I were to give this class
-->
without the virtual machine desktop,
-->
like the DAW desktop environment,
-->
you know, that we're working off of,
-->
that we, you know,
-->
we run into a few technical issues,
-->
you know, latency is a pain in the butt,
-->
you know.
-->
No, no, no, download and run it.
-->
Because it requires Java.
-->
Java is actually a bunch of jar files.
-->
And so.
-->
I mean, from the instructor's standpoint.
-->
Okay.
-->
You wouldn't, you wouldn't have any insight.
-->
You wouldn't be able to.
-->
Oh, that's true.
-->
You would have to share your screen
-->
if you have a problem.
-->
Okay.
-->
Yes.
-->
So, okay.
-->
So what I've gathered is,
-->
so the next training class,
-->
my understanding is,
-->
is they are not as technical.
-->
And so my goal for the next training class,
-->
we're going to focus just a lot on the flow building
-->
and the components of NAFA,
-->
not necessarily, you know,
-->
going into scalability,
-->
multi-tenancy,
-->
some of the minify,
-->
you know, how to build a custom processor,
-->
you know, those types of things
-->
that I'm going to kind of steer away from
-->
and really just focus on the NAFA canvas
-->
and building a flow.
-->
You know, Java is already installed on the,
-->
you know, we have on this virtual desktop
-->
the ability to install whatever.
-->
So I'm going to probably,
-->
you know, I on purpose did not install things for us
-->
because I heard there were sysadmins on the call
-->
that would help.
-->
But it's, I would probably be safer,
-->
you know, to give a better class next time,
-->
I'd probably be safer to have everything installed,
-->
everything going,
-->
you know, just log in
-->
and let's build like simple data flows
-->
and then graduate from there.
-->
Would that sound about right?
-->
Yeah, it does.
-->
Okay, okay.
-->
Yeah, I'm just reading a note from Maria
-->
that suggested that, you know,
-->
we could potentially use your own personal laptops
-->
and then I didn't know, I didn't, like,
-->
I didn't, I was thinking the same thing.
-->
It'd be hard to view what's going on
-->
because, you know, at least in this environment
-->
I can quickly see everybody doing work
-->
and what they're working on
-->
and those types of things.
-->
So, all right, we'll keep it in this environment.
-->
During the next training class,
-->
what if they run into the proxy to death issue?
-->
Will someone on that class know how to fix
-->
or should they, you know, reach out to one of you?
-->
Okay.
-->
Up front, before the class even starts.
-->
Oh.
-->
That's on, I don't even know who the right person
-->
would be to even ask that question.
-->
I know, I think one person will actually be
-->
100% taken care of though.
-->
Okay.
-->
Okay, okay.
-->
But he's on my team, that's how I know.
-->
Oh, okay.
-->
Good deal, good deal.
-->
No, I was, you know,
-->
I just want to make sure it's a little bit smoother.
-->
You know, unfortunately, you all are the guinea pigs
-->
who's getting, you know, the first of this class.
-->
So, I just want to make sure that the next class,
-->
I know I've heard is less technical
-->
and I want to make sure it's a little bit smoother for them.
-->
And then Maria had some concerns with the,
-->
you know, some of the DAW desktop,
-->
you know, and it's how latency can cause some major issues.
-->
I was getting 300 millisecond pain
-->
to this desktop environment, you know,
-->
and it just made for pointing and clicking very hard.
-->
And with NaFi, you have to, you know, click
-->
and bend, you know, bend these connections
-->
and put these flows down.
-->
So, okay, perfect.
-->
That answered my question.
-->
All right, so we are going to, you know,
-->
touch on some custom development
-->
and then go into, you know, some next steps
-->
that we need to do
-->
and then wrap up with any questions and answers.
-->
We'll probably get out of here a little bit early.
-->
If you want, we can actually go back to your data flow.
-->
Me personally, I love building data flows.
-->
I, you know, it is, I don't know what it is,
-->
but, you know, I don't have to open Eclipse.
-->
I don't have to open my IDE.
-->
I can build data flows.
-->
I can manipulate data.
-->
I can execute services, all those types of things.
-->
So, if you all want to, you know,
-->
kind of go through those, we can.
-->
If you want to call it a little early
-->
and go do your day job, that's fine as well.
-->
You know, so it's up to you.
-->
But, all right, so let's get into custom NaFi development.
-->
And I realize, you know, even on this call,
-->
you know, we have one, two, three, four, five devs.
-->
And so, you know, we're just going to touch on some things,
-->
you know, give you some pointers and some direction
-->
if you want to create your own processor.
-->
There is a ton of documentation out there
-->
to create your own custom NaFi processor.
-->
And I'm about to bring that up
-->
and point you in the right direction.
-->
But basically, if, you know, NaFi really leans on Java.
-->
And so, you know, as a developer,
-->
if you are familiar with, you know, Eclipse,
-->
you know, some of the other Java, you know,
-->
well, Eclipse handles C++ and everything else.
-->
I really like the new Microsoft IDE,
-->
you know, because of the Docker integration
-->
and, you know, Visual Code, you know,
-->
so I like it with the Docker integration,
-->
you know, the Java capabilities, those types of things.
-->
NaFi really makes it easy, you know, to get started.
-->
You know, so if you are getting started
-->
with developing a NaFi flow, you know,
-->
you would use Maven, of course, Java, of course.
-->
And your favorite IDE, I would recommend Eclipse or Visual Code
-->
because I feel like, you know, those two are,
-->
you know, the most supported.
-->
You can use IntelliJ as well.
-->
You know, that's another, you know,
-->
that's another one that I see quite a bit,
-->
you know, and but most likely now
-->
I see mostly Visual Studio, you know,
-->
things like that.
-->
Okay, so, you know, Apache NaFi, you know,
-->
has, you know, an Archetype Maven type in already.
-->
So if you were to, you know, use Maven,
-->
you can generate Archetype generate
-->
and, you know, search for the NaFi type
-->
in the Archetype projects.
-->
If you're familiar with that,
-->
you know, you should be able to see NaFi.
-->
You want to select, like the,
-->
and all of this will be in the slides,
-->
but you want to select the NaFi
-->
processor bundle Archetype project.
-->
You know, select the latest version of that.
-->
You know, just so, you know,
-->
for you non-developers, you can actually go to,
-->
you know, select, you can actually download
-->
these things as well.
-->
But anyway, so you want to select
-->
the NaFi processor bundle,
-->
and that would really kind of kickstart
-->
your project, right?
-->
And then once you do that,
-->
you can enter your group ID,
-->
artifact ID, your version,
-->
artifact base name, you know,
-->
some of these additional properties.
-->
And then Maven will create,
-->
will have created the following directory.
-->
So it's going to create a NaFi
-->
artifact base name processors,
-->
and a NaFi artifact base name
-->
NOR directory.
-->
And then from there,
-->
you just run, you know,
-->
open Eclipse, for instance,
-->
you know, Visual Studio will do it as well.
-->
But, you know, process those,
-->
those artifacts in those directories,
-->
do an MVM install,
-->
and you should have, you know,
-->
a project that loaded into your workspace
-->
with the, you know, artifact base name,
-->
you know, directory in Eclipse,
-->
for instance.
-->
You know, and then it's just,
-->
you know, adding code,
-->
and I'm working on bringing up
-->
a visual for this.
-->
Then it's just adding code,
-->
you know, so a lot of times
-->
what I've seen is, you know,
-->
just remember a processor
-->
does a single function.
-->
And so, you know, if,
-->
if you were to have already,
-->
I think you guys mentioned
-->
you may have a processor,
-->
you know, a Java jar already.
-->
I think converting that to a Java
-->
NAR would be, you know,
-->
pretty easily, you know,
-->
those types of things.
-->
Some of the other developer resources
-->
you have, again,
-->
I would never, you know,
-->
not anymore, right?
-->
You know, they say every good
-->
developer copies code, you know.
-->
So one of the things I like to do
-->
is go to GitHub.
-->
You go to GitHub
-->
and just type in not hot processor.
-->
You're going to get a ton of results.
-->
You know, there's examples of,
-->
you know, even InfluxDB,
-->
which is a, you know,
-->
a commercial company as well as,
-->
you know, they have the open
-->
source InfluxDB.
-->
They provide their own processor.
-->
Hadoop provides its own processor.
-->
GeoMesa, which is a,
-->
you know, a Geo database
-->
that, you know, you store,
-->
you know, map data
-->
and those types of things.
-->
It has its own processor.
-->
So there are tons and tons
-->
of processors out there.
-->
People creating new processors
-->
all the time.
-->
And like I said,
-->
if you just search
-->
for not hot processor on GitHub,
-->
you're going to,
-->
you know, get a ton.
-->
You know, if you have a connection
-->
to, you know, something
-->
that was the one I just saw you,
-->
I think yesterday,
-->
even a Terraform project
-->
for NIFI, some NIFI Docker,
-->
that's not software engineering,
-->
but, you know,
-->
there's a lot of resources.
-->
So when you are custom
-->
developing your own processor,
-->
please just go out
-->
and search GitHub
-->
and, you know,
-->
you will find the source code
-->
probably for what you're trying
-->
to make a connection to.
-->
There's also, you know,
-->
this one here is used quite a bit
-->
is a Modbus library
-->
that helps with the OPC UA.
-->
So if you're developing
-->
like an OPC UA processor,
-->
you know, there's processors
-->
built for that.
-->
You know, there's a lot.
-->
And so what I like to do
-->
when I used to do
-->
NIFI processor development,
-->
I would just go,
-->
and search and to see
-->
if somebody's already developed that
-->
processor that meets my needs.
-->
You know, in the past
-->
we had to develop
-->
our own InfluxDB processor,
-->
for instance.
-->
Now, you know,
-->
it's already developed,
-->
it's already supported
-->
by the organization.
-->
You know, so definitely
-->
take a look at that.
-->
I didn't do a whole lot on,
-->
you know, I don't think
-->
that a lot of people
-->
would be interested in a
-->
Java IDE class on this.
-->
But, you know,
-->
I just wanted to make sure
-->
I went over, you know,
-->
some of those types of things.
-->
Again, there's a lot of
-->
capabilities out there.
-->
There's a lot of documentation.
-->
Once you, you know,
-->
start looking at the
-->
official NIFI documentation
-->
on building a processor,
-->
they give examples
-->
and how to deploy
-->
and all that.
-->
We've talked about
-->
how to deploy,
-->
you know, a custom
-->
NIFI processor.
-->
You put it in the
-->
Extensions folder
-->
and go from there.
-->
I will send out
-->
a full write-up
-->
on how to build a processor,
-->
you know, some of the
-->
processor properties
-->
and everything else.
-->
I have a really,
-->
really good example.
-->
But, like I said,
-->
don't think folks
-->
would just want to sit back
-->
and watch me code.
-->
And it might be
-->
a little messy as well.
-->
So I will send that out
-->
to those that are interested.
-->
It will include
-->
an example processor
-->
as well as some of the...
-->
I'm going to make sure
-->
I incorporate some of the
-->
new features of NIFI
-->
so you can, you know,
-->
determine if that's
-->
a restricted processor or not
-->
and how you would do that.
-->
But I wanted to make sure
-->
I go over how, you know,
-->
it's a very well-supported...
-->
There's a lot of documentation.
-->
There's a lot of documentation,
-->
especially from NIFI
-->
and those types of things.
-->
So that's my touching
-->
on the, you know,
-->
software developing
-->
of a custom processor
-->
and going from there.
-->
If you have any questions
-->
specifically I can answer,
-->
you know, I'll be happy to answer.
-->
But I felt like,
-->
you know, we could code
-->
and then, you know,
-->
I don't have to ask
-->
the rest of the class to code
-->
and it would be, you know,
-->
a lot easier just to send this out
-->
and then if you're interested
-->
in creating custom processors,
-->
have at it.
-->
Again, make sure you check Github.
-->
I know most of the bigger,
-->
like Apache-level foundation
-->
software like Hadoop,
-->
Kafka, those types of things,
-->
they all have supported processors.
-->
Some of the bigger systems
-->
like Azure and AWS
-->
have processors as well.
-->
I think I mentioned
-->
some of the, you know,
-->
processors you get with,
-->
you know, and some of the capabilities
-->
you get now with Azure.
-->
And so I'll send that
-->
documentation out as well.
-->
And then if they're,
-->
you know, real quickly,
-->
if you have a question about,
-->
you know, developing a custom processor,
-->
you know, feel free to interrupt me.
-->
Some of the other things I've seen,
-->
you know, that you want
-->
to take consideration is this.
-->
And I mentioned it a couple of times.
-->
But, you know,
-->
this best practices for configuration,
-->
please take a look at this.
-->
I've seen many, many,
-->
many non-FI installs
-->
and they start running into,
-->
you know, lots of problems
-->
and they can't figure it out
-->
because, you know,
-->
they don't have the right configuration
-->
for file handlers
-->
and fork processes
-->
and those types of things.
-->
You know, so, you know,
-->
you want to increase,
-->
I think the default level
-->
in Ubuntu is 10,000
-->
or maybe even 1,000.
-->
But you want to increase
-->
the maximum file handlers that you have.
-->
You can go into your security,
-->
your limits.conf file
-->
and, you know,
-->
right here we have 50,000.
-->
But, you know,
-->
if you are processing
-->
a lot of small files,
-->
you may want to increase this to,
-->
you know, 75 or 100,000.
-->
NAFA also likes to fork
-->
a significant amount of processes.
-->
If you remember,
-->
when we were going through the Canvas,
-->
you know, we were setting,
-->
here is how many processes we can run.
-->
Let me pull this up.
-->
Perfect.
-->
How many concurrent tasks, right?
-->
You know, so as you can imagine,
-->
you know, we have one concurrent task
-->
per processor.
-->
But, you know,
-->
with that being said,
-->
you may want to increase that.
-->
We are using the bare minimum,
-->
you know, that we would need,
-->
you know, for this flow.
-->
But if you're ingesting
-->
a large amount of data,
-->
you want to, you know,
-->
increase that, you know,
-->
concurrent task
-->
because then how many they can run.
-->
When you start doing that,
-->
you're going to start,
-->
you know,
-->
hitting up against this max fork processes.
-->
So for you sysadmins out there,
-->
you know, definitely,
-->
if you haven't looked at this
-->
and make sure it was configured,
-->
please do.
-->
You're going to run into an issue
-->
in production if you don't.
-->
And, you know, a lot of times,
-->
you know, we'll start trying to trace it
-->
and trying to find out the problem.
-->
And, you know, we keep thinking
-->
it's not five, it's not five, it's not five.
-->
But it's actually, you know,
-->
some of the underlying configuration
-->
that needs to happen to the,
-->
you know, host OS.
-->
If you are, you know,
-->
dealing with a lot of minify
-->
and a lot of minify agents,
-->
you definitely want to increase
-->
the number of TCP socket ports available.
-->
You know, that is,
-->
again, I've seen this numerous times
-->
where, you know,
-->
those need to be increased.
-->
You know, if you're,
-->
if you're constantly setting up
-->
and tearing down a large number of sockets
-->
and a quick amount of time,
-->
you know, you'll need to increase those.
-->
And I've seen a lot of flows,
-->
you know, do that.
-->
But, you know,
-->
assist admin never came in
-->
and, you know,
-->
they just didn't know
-->
about the best practices.
-->
So, you know,
-->
this always gets touched on
-->
in every class I've ever seen.
-->
So I wanted to make sure I pointed it out.
-->
You know,
-->
NAFI has that, you know,
-->
data delivery guarantee.
-->
It's going to work over, you know,
-->
any type of network connection,
-->
including like Donald Up and others.
-->
So you want to set how long the socket
-->
stay in a time-to-wait state when closed.
-->
You don't want your sockets to sit
-->
and when you're too long.
-->
You know, so, you know,
-->
make sure in some of the later,
-->
maybe even past,
-->
I mean, some of the later kernels,
-->
you may not even need this.
-->
But if you're running a,
-->
you know, 3.0 kernel,
-->
I think Ubuntu has even passed that now.
-->
But I don't, you know,
-->
if you're using a CentOS or,
-->
you know, a lot of times the government
-->
will approve like a Red Hat version
-->
and Red Hat's like 10 years old.
-->
You know, so just keep that in mind
-->
as a sysadmin.
-->
Make sure you tell Linux to never use,
-->
you never want NAFI to swap.
-->
NAFI is not good at swapping.
-->
It's always wanting to run.
-->
It's always, you know,
-->
those processors are always running.
-->
You know, so turn off swappiness
-->
if you can, you know,
-->
in those types of things.
-->
There is, so just for FYI,
-->
there is a Clam AV processor.
-->
If you search GitHub,
-->
I think it's there.
-->
So you can scan, you know,
-->
all of the files that you're ingesting
-->
for antivirus, you know,
-->
for viruses using Clam AV and some others.
-->
But, you know, you may want to take a look
-->
at excluding the content repository,
-->
the flow file repository,
-->
logs, provenance, and state
-->
from your, you know,
-->
your antivirus scanning.
-->
It can take a long time
-->
to scan those directories.
-->
And, you know, it's just using resources.
-->
And those directories are going to get large.
-->
And so, you know,
-->
some of the best practices here
-->
is just, you know,
-->
keep working on, you know,
-->
exclude your antivirus.
-->
Those directories are going to get huge.
-->
Make sure that the, you know,
-->
the respective repository
-->
gets onto a good storage mechanism.
-->
We talked a little bit about how,
-->
you know, you may keep
-->
your content repository
-->
on something really fast
-->
because that's where,
-->
you know, the content of a flow file
-->
is getting written and read
-->
and those types of things.
-->
The provenance repository is,
-->
you know, a very fast, quick,
-->
you know, all of that's being written
-->
in your flow file repository.
-->
So, you know,
-->
your other repositories
-->
can stay on a, you know,
-->
a slower disk if you need to scan it
-->
with your antivirus.
-->
But if you scan these,
-->
they can get rather large.
-->
So, you know,
-->
it's totally y'all's call.
-->
But, you know,
-->
if you can get away
-->
with not scanning that repository,
-->
have at it, you know,
-->
that does cause some concern
-->
because, you know,
-->
what if somebody brings a virus
-->
into the system, you know,
-->
so take that into consideration.
-->
Also taking into consideration
-->
there's, you know,
-->
processors that will interact with,
-->
you know, and send files
-->
to an antivirus to check
-->
and those types of things.
-->
This was asked,
-->
and I wanted to make sure
-->
I pointed this out.
-->
NaFi uses LogBack now
-->
as its runtime logging implementation.
-->
They got rid of Log4j.
-->
Now, with that being said,
-->
Log4j can still exist
-->
in an IA processor.
-->
So if you download a processor
-->
that is not, you know,
-->
like officially supported
-->
from the Apache Foundation
-->
or NaFi, you know,
-->
it may have a Log4j element.
-->
But, you know,
-->
to enter the earlier question,
-->
I think it was on day one,
-->
NaFi uses LogBack now
-->
for many reasons.
-->
And so, you know,
-->
just keep in mind,
-->
you know, that capability.
-->
As security incidents happen
-->
and are reported,
-->
those are quickly fixed
-->
and new releases are sent out.
-->
I don't know
-->
if there's a process
-->
I've seen in the past
-->
where, you know,
-->
if NaFi does find a vulnerability
-->
and they fix it
-->
and they release a new update
-->
that someone is notified.
-->
So I would highly recommend,
-->
you know, checking in
-->
on the NaFi website.
-->
We've already seen just this week
-->
we went from 125 to 126.
-->
And so 126 came out yesterday
-->
or day before yesterday.
-->
And so, you know,
-->
just kind of keep an eye
-->
on the downloads page.
-->
You may need to,
-->
you know, reference that.
-->
If you are looking
-->
at the documentation
-->
and doing some of these
-->
more advanced things,
-->
you know, you'll probably see it.
-->
Again, in the documentation,
-->
you're going to mostly look
-->
at the NaFi app log.
-->
That is the log containing,
-->
you know, framework
-->
and component messages.
-->
Your bootstrap
-->
and some of the others,
-->
they don't get used as much.
-->
There is a user log
-->
and a request log
-->
pending on, you know,
-->
your adminified usage
-->
as well as your,
-->
you use the Python API software,
-->
you know, depending on that,
-->
you may see a lot of messages
-->
here as well.
-->
So definitely, you know,
-->
keep that in mind.
-->
Security is the hardest thing
-->
to set up with NaFi
-->
just because it's a lot
-->
of hand jamming.
-->
It's a lot of trial and error.
-->
There is a ton of configuration.
-->
You know, Cloudera makes,
-->
you know, a version
-->
that the Cloudera data management
-->
where they have a GUI for this,
-->
you know, that helps set this up.
-->
Unfortunately, there is not
-->
an open source GUI
-->
or easy way to do this.
-->
You know, so I think
-->
the question was about,
-->
you know, some of the policies earlier
-->
and, you know, just got to hand jam
-->
that initial configuration
-->
and initial policies in place.
-->
The answer is yes, unfortunately.
-->
So, you know, that's the downside
-->
to open source
-->
and that's a downside to NaFi
-->
is, you know, some of these
-->
more advanced configurations
-->
and those types of things,
-->
you know, you would take a look.
-->
Use the toolkit,
-->
the NaFi toolkit,
-->
the Minify toolkit.
-->
There is a lot of capability
-->
there to manage
-->
users and clusters
-->
and those types of things,
-->
you know, so definitely download those.
-->
They were downloaded
-->
on the instance that we were using.
-->
We didn't really need it
-->
because, you know,
-->
we worked on what we had,
-->
but I could have used it
-->
to manage the install.
-->
I could have used it to,
-->
you know, some of those things.
-->
So just make sure you have the toolkit
-->
and the Minify toolkit,
-->
you know, downloaded.
-->
It will help out tremendously.
-->
You will use it a lot.
-->
It's still used quite a bit.
-->
Now, it's all command line.
-->
There's no UI.
-->
So, you know, just bear with it.
-->
It's the best at the end.
-->
But yeah, let's see.
-->
What else?
-->
You have Apache Knox,
-->
Apache Ranger out there.
-->
You know, if your key cloak
-->
is another good one.
-->
I like key cloak.
-->
I just put key cloak in with NaFAT
-->
on a solution for the government.
-->
You know, so if you need,
-->
you know, some of those capabilities,
-->
I highly recommend key cloak.
-->
It's already ATO'd.
-->
It's already good to go
-->
on the government side of the house.
-->
So, you know, getting a liking time
-->
and getting some of those approvals,
-->
you know, kind of kick-started,
-->
would definitely help.
-->
Let's see.
-->
What else can I go over?
-->
You know, the multi-tenant authorization.
-->
You know, talking about the authorizers
-->
and those types of things.
-->
I'll include those links.
-->
But, you know, I know
-->
you're having an issue there.
-->
You know, a lot of that's going to be,
-->
you know, coming back to this.
-->
What else?
-->
Is there any,
-->
is there any, like, topic, discussion,
-->
anything that we can go over
-->
while we have our few minutes here left?
-->
Because I want to make sure
-->
I got all the questions.
-->
If I don't get them answered,
-->
I want to make sure
-->
at least get them documented
-->
so I can go back, answer them,
-->
and email this out.
-->
Quick question on the jar files.
-->
So, if you get them from GitHub
-->
and then you install it
-->
just through Maven,
-->
and then your clips and everything,
-->
then how would you
-->
add them back to the processor?
-->
Like, you import them or how does that work?
-->
Oh, great question.
-->
I'm really glad
-->
because I kind of highlighted that,
-->
but I didn't get to use it.
-->
Let me see if I can find a Nafaanar.
-->
Oh, Nafaanar.
-->
Does this guy have a release?
-->
No.
-->
Let me see if I can find a release.
-->
I don't want a Korean version.
-->
Let's see.
-->
If I can find a NAR to download,
-->
I can show you immediately how that...
-->
Let me see.
-->
That's a good question, though.
-->
Because I can show you exactly how you...
-->
Oh, I can use the GeoMesa, I bet.
-->
Yep.
-->
Releases.
-->
There's a NAR.
-->
Let's see here.
-->
GeoMesa, Datastore, Service, FileSystem, Kafka.
-->
Installer, Bundle, though.
-->
Oh, Influx.
-->
Perfect.
-->
Great.
-->
Great question.
-->
Let me address.
-->
So, I have that NAR, right?
-->
And so, I've developed my own custom processor.
-->
I have used Maven to build it and all those fun things.
-->
Let me...
-->
Done.
-->
Perfect.
-->
So, you've built your NAR.
-->
You've got it running.
-->
You've downloaded a NAR.
-->
But be careful what you download, right?
-->
So, what I like to do...
-->
Talk about this is...
-->
Here's the InfluxDB NAR.
-->
So, what I do is...
-->
If you remember, I can stop NAFA,
-->
and I can put it in the lib directory and start NAFA again,
-->
and it will show up in my processors.
-->
Best thing to do is extensions.
-->
So, you know, even if...
-->
Go speed racer.
-->
Even if, you know, you have a custom processor,
-->
I still recommend putting it into the extensions directory.
-->
I would leave the lib directory to your core supported NAFA,
-->
you know, NARs and libraries.
-->
And so, you know, any custom processors,
-->
you can have that go through a CICD process, build it,
-->
and you want to send it to the extensions directory.
-->
The extensions directory was specifically built,
-->
you know, to allow for custom processors
-->
and also hot deployment.
-->
So, if I go back to my NAFA instance...
-->
I haven't had to stop my NAFA instance.
-->
I haven't had to, you know, do anything.
-->
And now I should have an influx.
-->
There it is.
-->
So, now you can see I have an influx processor,
-->
I have an influx, get influx database record,
-->
put influx database, you know, those types of things.
-->
So, because I just took that NAR, downloaded it,
-->
put it into the extensions directory,
-->
it hot deployed that NAR, and now I'll have full access to it.
-->
Is that your question, Pedro, or was it something else?
-->
Yep, yep, very nice.
-->
That's what I was wondering.
-->
Perfect, I got one right.
-->
Okay, no, great question.
-->
And I mentioned the extensions directory a few times now,
-->
so I'm glad you asked how you do it.
-->
So, yeah, you know, you just take the NAR
-->
and put it into the extensions directory,
-->
hit refresh on NAFA, and it should, you know,
-->
it might take a minute.
-->
If you look at the logs, you know,
-->
usually when I'm sys-admitting, like, a cluster,
-->
I have putty pulled up in about 1,000 logs.
-->
But if you look at the logs, you can see, you know,
-->
it, that NAR getting deployed.
-->
And sometimes it just takes a minute to,
-->
well, I'm getting all kinds of, oh,
-->
my warrants is because of that connection.
-->
But anyways, you know, if you're reading your logs,
-->
you should be able to see, you know,
-->
that that NAR was deployed.
-->
There we go, NAR autoloader.
-->
So, you know, it autoloaded the NAR and deployed it.
-->
It's good to go.
-->
If, you know, sometimes it's not easily apparent,
-->
but you may want to, you know, as a sys-admin,
-->
you may want to tell this log.
-->
And when you're deploying into the extensions directory,
-->
just kind of watch things.
-->
The log, you know, the log file will have additional data
-->
that, you know, NIFI may not even,
-->
NIFI may not even report any issue with that processor
-->
until it's being used.
-->
And so, you know, you want to check your logs
-->
when you're loading custom NARs
-->
just to make sure that it deployed correctly.
-->
But great question.
-->
And I'm glad we got to do a hands-on,
-->
real-world here it is.
-->
Any other questions?
-->
Absolutely.
-->
So, that is not a policy in NAR.
-->
And so, you know, as a sys-admin,
-->
you need to be able to lock these directories down.
-->
And so, the only way someone should be able to
-->
bring in a custom processor is if they have access
-->
to the lib directory or the extensions directory.
-->
Now, you know, you can, you know,
-->
lock that directory down just to only root.
-->
For instance, depending on your security mechanism,
-->
there's some other capabilities there.
-->
But, you know, in NIFI itself, you know,
-->
you notice that we never went to NIFI
-->
when we're loading a custom processor.
-->
We just dropped it into the extensions directory
-->
and then NIFI loaded it.
-->
So, your way of securing that is you need to secure
-->
the extensions directory, which is outside of the NIFI UI.
-->
So, great question, though.
-->
All right.
-->
Other questions before we wrap this up?
-->
Well, a few housekeeping things.
-->
You will be, you either have been sent
-->
or you're getting sent a survey.
-->
I noticed I did say um a few times
-->
and I'm trying to not do that.
-->
So, hopefully, that didn't bother anyone.
-->
I did get a mark on that before.
-->
But, you know, complete your survey as soon as you can.
-->
My pay is tied to that survey.
-->
You know, just the timeliness of it.
-->
I get paid either way.
-->
But, you know, the quicker those surveys are completed,
-->
the quicker I get paid, you know.
-->
So, if I get, you know, five, six, seven
-->
of these surveys complete, I'm good to go.
-->
I will, I have, I think everyone's email address
-->
that's signed up.
-->
What I will be working on over the next, you know,
-->
tonight and tomorrow, I'm gonna work on getting
-->
all your questions fully answered.
-->
I'm even going to get a tip on how to rename the files.
-->
Because I know we ran into that.
-->
And so, I'm gonna start compiling all of that.
-->
And then I will email a PDF that will, you know,
-->
have all of this information as well as some tips, tricks,
-->
some of the questions you all asked.
-->
And as well as anything else I can do to answer.
-->
If you have any future questions, let me know.
-->
You know, technically, after this training's done,
-->
I'm done, but I never like to treat, you know,
-->
people like that.
-->
So, if you have a question, especially on NAFA,
-->
I really enjoy this because it's one of my,
-->
quote unquote, babies.
-->
You know, feel free to send the question over.
-->
I'm also gonna send some YouTube links.
-->
There's some really cool design patterns that,
-->
you know, that we kind of went over.
-->
But we just don't have the time to fully,
-->
you know, go through every design pattern.
-->
But my friend Mark Payne puts these together on YouTube.
-->
And so, I'm gonna send them over to you.
-->
And yeah, if you have any other additional questions
-->
or anything else, shoot me a note.
-->
If it's something short and quick, I'll be happy to help.
-->
If you got more long-term, you know, support
-->
or something like that, you know, feel free to let me know.
-->
Like I said, I contract to this training company
-->
through my data engineering firm.
-->
So, you know, I'd be happy to help out wherever I can.
-->
And if there's no additional questions,
-->
I'd give you back a few minutes of your time
-->
and have a good rest of the day.
-->
Thanks, everyone.
-->
Have a good one.
-->
Why are you so cute?
-->
Because you have small hands, dude.
-->
I'm in there, that's why.
-->
I'm in there.
-->
Oh, are you having fun?
-->
You're doing good, that's fine.
-->
I'm having a blast, Dad.
-->
Yes?
-->
I'm having a blast, Dad.
-->
Oh, no.
-->
What is it?
-->
I don't know.
-->
It might be a little bit crazy, like that.
-->
Ah!
-->
Not even 90% at all!
-->
Hey!
-->
Oh, this is really, of course it's fucking good!
-->
Like, is it really or not?
-->
Maybe, and the other one that's like,
-->
is it a bowl or a foot over it?
-->
Oh, a bowl or a foot over it.
-->
Uh-huh.
-->
Oh, wow.
-->
Oh, I love that, it's like,
-->
you're good.
-->
You're good.
-->
Dad, are you vlogging?
-->
I'm vlogging.
-->
I'm vlogging.
-->
She did it!
-->
She did it!
-->
She did it!
-->
She did it!
-->
It's up.
-->
It's up.
-->
Do you want it, Dad?
-->
It's up.
-->
It's up.
-->
It's up.
-->
It's up.
-->
It's up.
-->
It's up.
-->
Do you want to go eat somewhere tonight?
-->
Do you want to go to the grocery store?
-->
Do you want...
-->
I can't remember the name, but...
-->
Okay.
-->
This is sweet, right?
-->
Yeah.
-->
Okay.
-->
Oh shit!
-->
Oh!
-->
Urine?
-->
Oh, my God!
-->
Let's see how it goes.
-->
I don't know what it is.
-->
Hey!
-->
What's the rating?
-->
Uh, 30% came down, 57% came in.
-->
11% came, uh, of the medium.
-->
57% large, 43%, 42%.
-->
123% are protected, and 20% are protected.
-->
527% are protected, and 20% are protected.
-->
So what's the overall rating?
-->
50% yes, and 110%.
-->
I'm going to show you all my list right now.
-->
1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15.
-->
I've got a lot on my phone already.
-->
Let me guess.
-->
I'm like, I just got a pack of cigarettes in my car.
-->
That's red.
-->
I was like, you suck, dude.
-->
I don't know what to agree with.
-->
What is this?
-->
Drugs.
-->
Drugs.
-->
Drugs.
-->
Oh.
-->
Anyway, socks on.
-->
Oh, socks got dirty.
-->
That's one fucking yard.
-->
We have one of them.
-->
One of them.
-->
I'm going to leave you.
-->
No.
-->
OK.
-->
Look at him.
-->
Look at him.
-->
Look at him.
-->
Look at him.
-->
Look at him.
-->
Look at him.
-->
What's the one blue there, too?
-->
No.
-->
No.
-->
No.
-->
No.
-->
No.
-->
Whoa.
-->
Whoa.
-->
Whoa.
-->
Oh
-->
Oh
-->
Oh
-->
Oh
-->
Oh
-->
Oh
-->
Oh
-->
Oh
-->
Oh
-->
Oh
-->
Don't take it
-->
Oh
-->
Oh
-->
Oh
-->
Oh
-->
Oh
-->
Oh
-->
Oh
-->
Oh
-->
Oh
-->
Oh
-->
Oh
-->
Oh
-->
Oh
-->
Oh
-->
Oh
-->
Oh
-->
Oh
-->
Oh
-->
Oh
-->
Oh
-->
Oh
-->
Oh
-->
Oh
-->
Oh
-->
You want to put it up
-->
What are you doing
-->
So I'm going to be like two inches
-->
Okay
-->
Don't make a man
-->
So
-->
Yeah
-->
Your friends
-->
But uh
-->
Okay
-->
Oh
-->
I was like look
-->
This guy over here
-->
I don't care
-->
So I think I'm gonna have to wait for that.
-->
I'm gonna wait here.
-->
I'm gonna go pee and then I'm ready.
-->
I'm gonna take photos first.
-->
I'm gonna take photos first.
-->
I'm gonna take photos first.
-->
I'm gonna take photos first.
-->
I'm gonna take photos first.
-->
I'm gonna go pee and then I'm gonna take a shower.
-->
I'm gonna take a shower.
-->
It's the last one we'll get it up.
-->
Thanks for watching!