39:11
2024-05-06 08:57:50
3:11:28
2024-05-06 10:30:19
24:39
2024-05-07 07:20:20
19:49
2024-05-07 08:03:25
1:14:35
2024-05-07 08:35:13
15:41
2024-05-07 10:06:25
38:33
2024-05-07 10:37:03
2:48
2024-05-07 11:19:01
59:37
2024-05-07 11:33:56
6:10
2024-05-07 14:55:25
39:40
2024-05-07 15:02:44
9:45
2024-05-08 06:44:21
29:27
2024-05-08 08:09:24
2:51:40
2024-05-08 12:09:24
Visit the Apache Nifi - GROUP 1 course recordings page
WEBVTT
-->
Alright, we'll just go around the room. Sean, how are you doing?
-->
Mostly okay. I was kind of doing some catch-up from this morning still too, but I got it.
-->
Converting all the CSDs into JSON and putting them into a separate directory.
-->
And I was just trying to figure out how to merge them right now.
-->
Oh, very nice. You can pick them back up.
-->
Instead of writing the JSON, you might want to just send it to the next processor.
-->
But if you want to write it to disk and then turn around and pick those back up
-->
and then look at what Pedro and I were just going over, the EvaluateJSON processor,
-->
you can then... Okay, okay. So I think you're very close. You've got the CSVs and the JSON all
-->
aligned. It's now just the final calculations. So you're very close. Okay, good deal. Any
-->
questions I can answer? No, I don't think so.
-->
Okay. Alright. How about Aaron? Aaron, how are you doing?
-->
I'm going. Are you in the room with everyone else?
-->
Yeah. I am very surprised that the room is not complete. I figured you guys would be just
-->
sharing information constantly. So we are.
-->
Okay, good. But as far as we can get is making sure they're all in JSON format.
-->
As far as we've got, well, some of us are best with merging right now.
-->
Okay, okay. So again, you may want to look at the EvaluateJSON processor. Now that you have
-->
everything in JSON, you can write it to disk. If that's easier, pick it back up and look at
-->
that and look at taking those JSON values and pushing them up as an attribute. But it's
-->
funny as I was thinking, I was like, you know, there's three, four, five people in
-->
the same room. Like it would be so easy just to copy off of each other and be done.
-->
Yeah, we're trying. No, there's no, you know,
-->
nobody's gonna get dinged for helping your buddy, right?
-->
Okay, so who else is in the room? We have Brett in the room, I think.
-->
No, Alyssa. Okay, let's go to Alyssa. Alyssa, how are things with you?
-->
I got to the merge JSON. Okay.
-->
I haven't looked into the EvaluateJSON processor yet, but I basically merged all the files
-->
together. It doesn't look good, but they're merged.
-->
Don't worry. If you want to beautify the JSON, do remember we have, you know,
-->
if you bring down processor and you start typing JSON, there is numerous JSON processors.
-->
Some of those are used to beautify JSON. So, you know, I think it's, go down just
-->
a little bit. There's the flatten JSON that helps. EvaluateJSON is one that you will probably use.
-->
Scroll on down. Keep going. There's a jolt transform JSON. So, I'm surprised no one's
-->
kind of looked at that processor yet because you can actually write a jolt transformation
-->
and probably accomplish this task in four or five processors. So, you know, definitely take a look
-->
at the jolt transform JSON. Take a look at the EvaluateJSON path. I think you're getting close,
-->
right? So, yeah. I was thinking, you know, somebody in that room is going to write a jolt
-->
transformer and just have all this working within two or three processors. But no,
-->
I think you're getting close. Any questions I can answer right now? I think you're on a good
-->
path, but I just want to check in and see. Yeah, I think we all just need to play around
-->
with it a bit more. Like, we didn't find all the processors. There's over 300 processors. And so,
-->
make sure when you go into your processors, look for JSON processors. Look for CSV processors.
-->
Look for the items that you're trying to do. And you'll start narrowing down your list.
-->
Even as someone like myself that is extremely experienced at building data flows and
-->
9.5 processors and those, I will, you know, when I go look at, you know, when I have a
-->
problem, I'll go to my processors. I'll do a keyword like JSON. Let me see what's out there.
-->
Let me see what's available. You know, what are some of the capabilities? I keep the
-->
9.5 documentation up. And that way it has the documentation as well. One tidbit of information
-->
on the evaluate JSON path. You're only able to extract one field if you're trying to do a
-->
flow file content. So make sure you have flow file attribute so you can extract all the
-->
processor. But I think you're getting close. I would definitely look at the evaluate JSON path
-->
and they don't transform depending on your skill level and your skill set. You know,
-->
you might be able to solve some of this pretty quickly. Any questions that I can answer for
-->
Lisa? No, I would just need more time with her. Okay. Like I said, you know, take all the
-->
time. Like training stops at five, but I don't stop. I mean, I keep going, whatever it is to
-->
help support you all. So, you know, just keep going. What we'll do is we'll start in the
-->
flows. And even if you do not complete your flow, you know, if you at least have, you know,
-->
here's how I want to design this and here's how I think this might work. I'll take that as a
-->
answer as well. I would like to see this completed. But also, you know, we have
-->
different mix of technologies and things like that in the room. So, you know, it's
-->
if we just take it complete. So, okay, good. Who else do we have in the room with you all?
-->
Just me, Amanda. I didn't really get far. I just need to play around a little more with it.
-->
Okay. Is there anything really like you're hung up on or stumping you?
-->
I just don't know how a lot of these processes work, honestly.
-->
Okay. Because at first, like one of them, I was like trying to
-->
change the data and then trying to split it by like commons with these on attribute
-->
that I try to split the text, but that didn't work. I did. I tried.
-->
No worries. No worries. So for you, you want to get the weather data. If you, again,
-->
if you want to solve this very pretty quickly, reuse the CSV processors that we did for the
-->
previous exercise. And then you can actually write that JSON back to disk if you want
-->
and then have a new processor that grabs all your JSON. So it would include that station,
-->
that third station ID. And then you can bring all your JSON in together.
-->
You can evaluate the JSON path, extract the data, and then you can manipulate it using
-->
attributes. So just play around with it. If you don't get complete, no big deal,
-->
we can work on it. If you need to reference the previous exercise, you can upload that.
-->
It's in that zip file. I just didn't advertise it at first because I wanted to walk us through the
-->
flow itself. But you're able to bring down a process group, import that flow, and it
-->
should work for you. So if you need to reference the previous flow and even if you
-->
didn't get it finished, you can import it and it's complete. And that should give you some
-->
helpful tips on processing that CSV that you get. Any questions Amanda I can answer right now?
-->
Oh, no. I'm good. Okay. Well, try your best. And like I said, any questions, just
-->
feel free to reach out. And is there anybody else in the room with you?
-->
Hi, I'm Ben. I hope you like to scroll. Who was that, Cody? Ben. Oh, yeah. I was going to say
-->
Ben should be in there. Do I like to scroll? Oh, Lord. What's spider web? Oh, my God.
-->
Did you read Charlotte's Web before this? Well, what's the next thing I have to do?
-->
I have no clue where to start. Is this like a maze?
-->
Top left. So you're getting the source file. Oh, you identified the mom type.
-->
All right. So if it is a JSON document, you're sending it to write JSON to work. If it is a
-->
CSV, you're converting the record, and you're putting the file to JSON, just like we were
-->
doing on the first, you know, the previous exercise. And then you're pushing everything
-->
to a funnel and logging. Okay, so what's on the right? So I see your work folder.
-->
Okay, so it gets work files. Oh, okay. And then you merge JSON data, rename the file,
-->
you put the file again. Okay, perfect. Well, the nice thing is, I think you're very close
-->
on normalizing all of your data. Okay, perfect. So look at picking that back up.
-->
Might as well put a few more files on there. You don't have enough. Go ahead and get the file
-->
and take a look at the evaluate JSON or the flatten JSON processor, the evaluate JSON processor,
-->
the jolt transformation processor. I bet if you were to Google, I want to jolt transform
-->
a JSON document, you will see tons of great examples. Okay. So I think you're very,
-->
very close. Do you have any questions I can answer right now? And if not,
-->
we will go over this in the morning. Okay. Thanks, Ben. That is a spider web. Tyler,
-->
how are things going with you? Oh, wow. Wow.
-->
Just in just the files, I check the MIME type, my update to the schema attribute,
-->
route on attributes. Otherwise, it will convert to JSON. Wow. That's nice. Nice and clean.
-->
Then it queries the records to do the aggregations and pushes out the aggregations here,
-->
but I haven't branched that part out yet. And then this query record does the warning for
-->
speed. I haven't added any other warnings yet. And then I have it splitting the records and then
-->
extracting out the date as an attribute and then I'll merge it all from the same date
-->
to aggregate them for the reporting. And then I haven't fleshed that out yet.
-->
Oh, good Lord. You're almost done. You nailed it. And I was very surprised to see query record,
-->
split record. I really like that approach because that approach is a data engineering
-->
thought process where you can then take this and build some of the reusable components
-->
and process data like this in the future. I think you're very, very close to being complete.
-->
Any questions that you may have that I can answer? Any major roadblocks in your way
-->
that I can answer? But overall, great job. I don't think so.
-->
I think you're very, very, very close. So great job. If you need anything, let me know,
-->
but wow. Nicely done. Thank you.
-->
All right. Brett. I think Brett's done.
-->
No. Can you hear me? I can.
-->
So I started one way and then I switched and I started using the evaluate JSON.
-->
Okay. So I have for the native JSON file in there, I'm able to grab the
-->
file and get all the attributes for all the elements in there. So then now I'm just,
-->
because I split, I had two different paths for CSV JSON. The CSV path was the old one that we
-->
tried to go to the success to do the same thing. I haven't tried that yet.
-->
Okay. It may require a different evaluate JSON processor just to, you know, depending on
-->
if the JSON is exactly the same or not. But you're extremely close and good job using
-->
the evaluate JSON processor and pulling all of that data out as an attribute. You have it all
-->
of the capabilities you have, you know, now with that data. Any questions I can answer?
-->
This is, you know, you and Tyler are just really rocking it. Any questions I can answer?
-->
No, I think it's making sense. Okay. Perfect. I think you're very,
-->
very close. So I've been struggling with the canvas and the latency that we have though.
-->
You and me both. If it was my way, we would have a different environment. But I am as well. I just
-->
got a warning that I'm 300 milliseconds. And, you know, I'm like, come on now.
-->
But no, I get it. I get it. Okay. Well, if you have any questions, let me know. Great job.
-->
We'll go over this in the morning, but I think you're extremely close.
-->
All right. I think we went over
-->
Alissa and Amanda. Randy. How are things going, Randy?
-->
Well, that doesn't sound good. Well, it sounds like Randy's allergies are really getting to him.
-->
It is that time of season. So we'll come back. Randy, if you have any questions or anything,
-->
let me know. But I'll check in on your flow when I can see it and see how far you've gotten.
-->
Okay. Let's see. Who did I miss? Erin, I think I got you, but did I miss you?
-->
No, you looked at Erin's earlier. That's what I thought. It's confusing
-->
me with somebody in the room. And then, so did I miss anyone? And if I did,
-->
let me know. If not, does anyone have any questions I can answer right now? Okay. So
-->
it's about 30 minutes after, 22 minutes after I am going to call it. It's almost 5.30 my time.
-->
We will start off tomorrow morning going through these flows. We have enough time in the morning
-->
to spend 30 minutes to an hour going through these or updating them or finishing them up.
-->
So do your best you can. Again, if you do not get complete, what I'm looking for is just
-->
an explanation, a story on how you're planning to design your data flow and some of the things
-->
there. Little tips. I would try to get it organized and get some of the labels and the
-->
stuff that we went over. You know, on the canvas as well. You know, beautify the flow.
-->
Try not to read Charlotte's Web tonight, Ben. And, you know, have a great evening.
-->
If you have any additional questions, you know, speak now. If not, I'm going to go
-->
ahead and leave the meeting. But I will also check in on everyone's desktop later tonight
-->
because I have Minify and Notify processor development to go over tomorrow. And I need
-->
to make sure you all have the files for that. Awesome. Well, thank you all very, very much.
-->
Continue working as you as you want and we will pick up tomorrow morning. I will see you all
-->
then. Have a good evening.
-->
I'm saying that I am not an expert, but I am the mother of a trans child.
-->
And the other thing is that I love the sensations of medication for the trans community,
-->
specifically mine or someone else's. So rolling that in brief, all I had ready
-->
as for me, started developing super light in fourth grade. And you could tell that she
-->
was sitting on her skin and it was warm over her. But here's the thing. By seventh grade,
-->
we all caught up and learned and enjoyed the best for childhood. My school told me so.
-->
So the next thing to learn and trans kid is that we don't have the same luxury of
-->
everyone else catching up. But everyone else is doing the same as us. So the anxiety
-->
level that is created by not allowing children to take these medications and take care of
-->
and grow into themselves is so strong that we actually may be hurting certain children.
-->
Existing governments say to theirs, the sexism of an eternal and pressing death
-->
chain tradition for the rest of their life. And if you try to hand-ize people into
-->
making them believe that they are happy in this death, you think they're successful?
-->
I describe to you just there exactly what is going on in the world today.
-->
Yes, there's the scene. There's the scene.
-->
I'm between myself, my closest friends.
-->
And then I can't draw a straight line.
-->
It seems kind of cool.
-->
No, of course I am.
-->
So I'm going to need you for a script.
-->
I've got 24 hours and I'm going to tell you something.
-->
It was a dangerous job every day.
-->
So yesterday's...
-->
I think you got it.
-->
...liberal woman.
-->
Look, nobody's coming for your support.
-->
Okay, what we really want is common sense of what's in control.
-->
That's this.
-->
It's hard.
-->
To make a motion for no reason at all.
-->
And doing it on this place where it's cold out, mostly green, roughly, is not a good idea.
-->
There's also a problem doing it when it's too hot outside.
-->
Every time that's happening, you're rising pretty above 100 degrees, for sure.
-->
For all the work that you're doing at Society of Iowa, it is a good idea.
-->
Yeah, of course.
-->
That's it.
-->
Okay.
-->
And the father of you would do the same.
-->
That sailor from Stormy Daniels has been on the witness stand for a couple hours.
-->
Ladies and gentlemen, as you're all about to snap on water, I just wanted to share a new sock set I just bought for you guys.
-->
It's a pretty good size, and I'm pretty happy to upload it.
-->
It goes all the way, it starts at 26 inches, and then it goes 26 by 20s, and then 20 by 10 degrees, and 26 by 10.
-->
It's also better to be able to move stuff.
-->
I would say it's like 300 to 500 meters.
-->
All you have to do is take it on in 500 meters.
-->
It's a pretty good size.
-->
So, I mean, you could probably set it up to like an adapter, if you wanted to use this for any traction or something.
-->
But pretty good stuff.
-->
Here are the top ten largest churches in the United States based on attendance, per week.
-->
According to AppReach 100 data, at number 10 is Fellowship Church in Great Valley, Texas.
-->
When I popped the hood of this 2011 Honda Odyssey, a lot of y'all said that it was fake.
-->
Those of you that knew it was real just wanted to hear it out.
-->
We're going to cover both topics right now.
-->
We're going to listen to this thing out, and we're going to take a better look at the hood and back story for those of y'all that missed it.
-->
We've only got a little bit of time left, and I did, because I think it was my game.
-->
I'll do it at number three.
-->
First, I'm going to turn it on.
-->
I'm going to turn it on.
-->
We're going to turn on number three and number four.
-->
I'm going to turn it on.
-->
I'm going to turn it on.
-->
We're going to turn it on.
-->
Oh, jeez.
-->
Oh, jeez.
-->
We're going to turn it on.
-->
Our head is in lock.
-->
I can't even do that for all of you, you have the freedom to do whatever you want.
-->
Give me all the sound into that song on my head.
-->
Give me all the sound into that song on my head.
-->
Give me all the sound into that song on my head.
-->
Give me all the sound into that song on my head.
-->
Give me all the sound into that song on my head.
-->
Give me all the sound into that song on my head.
-->
Give me all the sound into that song on my head.
-->
Give me all the sound into that song on my head.
-->
Give me all the sound into that song on my head.
-->
Give me all the sound into that song on my head.
-->
Give me all the sound into that song on my head.
-->
Give me all the sound into that song on my head.
-->
Give me all the sound into that song on my head.
-->
Give me all the sound into that song on my head.
-->
Give me all the sound into that song on my head.
-->
Give me all the sound into that song on my head.
-->
Give me all the sound into that song on my head.
-->
Give me all the sound into that song on my head.
-->
Give me all the sound into that song on my head.
-->
Give me all the sound into that song on my head.
-->
Give me all the sound into that song on my head.
-->
Give me all the sound into that song on my head.
-->
Give me all the sound into that song on my head.
-->
Give me all the sound into that song on my head.
-->
Give me all the sound into that song on my head.
-->
Give me all the sound into that song on my head.
-->
Give me all the sound into that song on my head.
-->
Give me all the sound into that song on my head.
-->
Give me all the sound into that song on my head.
-->
Give me all the sound into that song on my head.
-->
Give me all the sound into that song on my head.
-->
Give me all the sound into that song on my head.
-->
Give me all the sound into that song on my head.
-->
Give me all the sound into that song on my head.
-->
Give me all the sound into that song on my head.
-->
Give me all the sound into that song on my head.
-->
Give me all the sound into that song on my head.
-->
Give me all the sound into that song on my head.
-->
Give me all the sound into that song on my head.
-->
Give me all the sound into that song on my head.
-->
Give me all the sound into that song on my head.
-->
Give me all the sound into that song on my head.