Wouldn’t it be awesome if there was a language as elegant as Spring Boot is as a framework? In this episode of Streaming Audio, Tim Berglund talks with Josh Long, Spring developer advocate at VMware about Kotlin, about the productivity-focused language from our friends at JetBrains, and how it works with Spring Boot to make the experience leaner, cleaner, and easy to use.
Josh shares how the Spring and Kotlin teams have worked hard to make sure that Kotlin and Spring Boot are a first-class experience for all developers trying to get to production faster and safer.
They also talk about the issues that arise when wrapping one set of APIs with another, as often arises in the Spring Framework: when APIs should leak, when they should not, and how not to try to be a better Kafka Streams when the original is working well enough.
EPISODE LINKS
Wouldn’t it be awesome if there was a language as elegant as Spring Boot is as a framework? In this episode of Streaming Audio, Tim Berglund talks with Josh Long, Spring developer advocate at VMware about Kotlin, about the productivity-focused language from our friends at JetBrains, and how it works with Spring Boot to make the experience leaner, cleaner, and easy to use.
Josh shares how the Spring and Kotlin teams have worked hard to make sure that Kotlin and Spring Boot are a first-class experience for all developers trying to get to production faster and safer.
They also talk about the issues that arise when wrapping one set of APIs with another, as often arises in the Spring Framework: when APIs should leak, when they should not, and how not to try to be a better Kafka Streams when the original is working well enough.
EPISODE LINKS
Tim Berglund (00:00):
I was going to talk to Josh Long about Kafka and Spring Boot, and you know we eventually got there. We might have meandered through another topic or two on the way, like say child-rearing, how much we missed conferences and the quality of Josh's bear hugs. But we do talk about Spring Boot and Kafka, which is very much a chocolate and peanut butter kind of situation. Listen in on today's episode of Streaming Audio, a podcast about Kafka, Confluent, and the cloud.
Tim Berglund (00:31):
Hello, and welcome to another episode of Streaming Audio. I am, as always, your host, Tim Berglund, and I am joined in the studio today, the virtual studio, by my friend, Spring Developer Advocate at VMware, Josh Long. Josh, welcome to Streaming Audio.
Josh Long (00:48):
Hey! Hi, how are you? Thanks for having me!
Tim Berglund (00:51):
You bet.
Josh Long (00:51):
It's good to be in the virtual studio.
Tim Berglund (00:54):
Yes, the virtual studio, better known to the world as your condo and a room in my basement. The acoustics are decent, so we'll take it.
Josh Long (01:05):
Yeah, yours sounds great.
Tim Berglund (01:09):
Nobody needs to know.
Josh Long (01:11):
Nobody needs to know?
Tim Berglund (01:13):
Where we actually are, the humble, physical surroundings that we're actually in. But hey, it's in the pandemic, everybody is at home.
Josh Long (01:20):
Yeah. No, I was going to say, as happy as I am to talk to you, I am not just a little sad that of all the millions of times that we've bumped into each other just last year, and then the year before that and all the- [crosstalk 00:01:37]
Tim Berglund (01:36):
And the bear hug I got from you every single time.
Josh Long (01:40):
Right. It's just sad that now it's like we finally both of us have enough time in a pandemic to sit down ... of course, we have to social distance.
Tim Berglund (01:50):
Yeah. It's pretty good, it's a thousand miles here, it's pretty good social distancing.
Josh Long (02:02):
Yeah, we're really ... when we commit, we commit.
Tim Berglund (02:02):
We do, we've taken it seriously. Yeah, we were just talking before the show, it was 10 years ago in June that we met, at a conference called Epicenter in Ireland. And I remember that show so clearly, became friends with the organizer, Barry Alistair, he's a great guy still. He gets me into ... more frequently than I deserve. And my wife and kids were with us for that trip. That was when my oldest son turned 16, so that was a memorable one.
Josh Long (02:34):
Oh, wow, so that was a decade ago.
Tim Berglund (02:37):
It was a decade ago.
Josh Long (02:39):
Isn't that weird? I mean that kid has [inaudible 00:02:41] everything.
Tim Berglund (02:45):
He's 26 now. Yes, he's married, he's all grown up.
Josh Long (02:49):
What? So if you had guessed 10 years ago that things would ... If you could imagine what things would look like in 2020 for your family, not for- [crosstalk 00:03:01]?
Tim Berglund (03:02):
Yeah, I was just going to say, there's a whole bunch of stuff I wouldn't have called, but no, that's not ... 26, I mean, I know that seems young to a lot of people. That wouldn't have seemed too shocking. I wouldn't put a timer on it or anything, but I'm not surprised that he did.
Josh Long (03:18):
Well, he's closer to 30 than not, that's a good way to think about it.
Tim Berglund (03:20):
That is also true, and I am going to text him that when we're done with this podcast. "Zach, you are closer to 30 than 20, I just want you to rest in that fact."
Josh Long (03:30):
Wow, just marinate in the passage of time.
Tim Berglund (03:35):
Yeah. Actually it seems like we talked about this on the podcast recently. You don't really want to hold on to 16 as long as you can.
Josh Long (03:43):
No.
Tim Berglund (03:45):
In respectful disagreement with John Cougar Mellencamp, it's simply not what you want to do.
Josh Long (03:51):
My little girl, she's 15, and she's starting to realize that she can't always be our little girl. She's got to start growing into her future self.
Tim Berglund (04:08):
Right, right.
Josh Long (04:09):
And so every now and then, she gets very sad like, "I just want to stay a kid." Just like, "Aww." There're moments where she just comes out all melancholy, and realizes soon she'll be in university.
Tim Berglund (04:25):
Yeah. Apart from her parents. And that's ... if being at home and having the parents that she has is a good thing, then that is a little something to grieve, like not looking forward to being separate, but wow, let's do it.
Josh Long (04:38):
Of course. And then again she's also looking forward to, in four months or whatever, she'll be able to get her driver's permit, so she's like, "Well, I want to get old enough to do some things but not old enough to do others."
Tim Berglund (04:53):
It's complicated.
Josh Long (04:54):
It is. I feel for her. It's not easy.
Tim Berglund (04:58):
Everybody, if you're concerned, we're going to talk about Kafka and Spring Boot, that's coming. It's Josh and ... and it's been a while since we talked, so give us a second here. But my middle daughter, you might remember my middle daughter, I don't remember what birthday it was, and she'd probably prefer that I didn't even say that. She's 24 now, also married, she just got married in May. She's a COVID bride.
Josh Long (05:20):
Oh, congratulations.
Tim Berglund (05:21):
Thank you. But there was some birthday where the night before, she was in bed crying.
Josh Long (05:27):
Oh, no!
Tim Berglund (05:28):
Yeah. And I'm like, "Oh, what's wrong?" And she had it in mind that that next birthday was big girl, and that meant that she would have to stop playing with dolls. And she just couldn't countenance a life with no dolls. And I'm like, "No, that's not true. You can still play with dolls for as long as you want, really, and then you'll grow up and have kids and get to play with dolls with your kids."
Josh Long (05:54):
Yeah. And even better, I'm a comic book enthusiast, I always have been, and I used to ... One of the joys for me of getting into my own working life was suddenly I ... not only did I not stop playing with action figures every now and then, but the quality of the action figures I could afford increased markedly.
Tim Berglund (06:17):
Right?
Josh Long (06:18):
So getting older, getting a job, doesn't mean you have to stop playing with toys. It just means you can get really nice toys.
Tim Berglund (06:29):
Exactly, yeah.
Josh Long (06:30):
So, speaking of nice toys, you and I get to work in open source.
Tim Berglund (06:30):
We do. Thank you, excellent segue-man move there. We do.
Josh Long (06:37):
That you and I are ... you're one of my heroes, because ... you know that, but I'm telling the audience ... you're one of my heroes, because you've made the choice to work with awesome stuff. And every step of the way, I see you at different organizations that are doing amazing things, and I love where you are right now because I feel like this is one of the best organizations, bar none, full stop, out there, period.
Tim Berglund (07:07):
And then at the risk of sounding somewhat tendentious, I agree with that last part there. Yeah, go on.
Josh Long (07:15):
But I just love your deliberate focus on going where awesome is. And you and I have known each other, like we said, for 10 plus years now, and yet you've been at least in four different organizations in that time I think. How long have you been with Confluent now? It feels like you are as long as Kafka.
Tim Berglund (07:35):
Three and a half years. When we met, I was actually independent, just kind of doing stuff.
Josh Long (07:40):
GitHub.
Tim Berglund (07:42):
GitHub, and then there was-
Josh Long (07:43):
DataStax.
Tim Berglund (07:45):
DataStax, the Cassandra community.
Josh Long (07:49):
And then this. And I always trail you. I'm always learning. In 2010, I knew a little bit about Git, but a minute later it seems like you're out there just teaching people the absolute details of the structures behind Git and doing it while reciting Shakespeare. I'm like, "Oh! That's amazing." And then Cassandra, there are still videos you can bump into that are epic if you want to learn Cassandra and those bits. And now, I love your videos on Kafka and Confluent and the whole Confluent platform. So it is really cool to watch you, my friend, you're a great teacher, and it's just been a great privilege for me, because I'm always constantly trailing you like, "Oh, if he's learning this, I probably should pick it up, too?"
Tim Berglund (08:36):
Oh, thank you. Thank you, Josh. [crosstalk 00:08:39] I really appreciate that. We should probably talk about ... I want to talk about Spring Boot, and how it intersects with Kafka, and with the ... It's a chocolate and peanut butter story. I was just telling actually one of the producers of this podcast earlier today that in general the Spring Framework, plus all things Kafka, is a value-creating combination, like chocolate and peanut butter. But in the event that ... maybe this is silly ... but if you're in the Java world you probably know what Spring Boot is, but there are going to be listeners who don't, either new to the world, or new to Java, or something. So for those people, get us started. What's Spring Boot?
Josh Long (09:33):
Okay. So first of all, I regard this possibility with respect and reverence, because I've heard about it, and I regard it with the same due skepticism, but I can see that it's plausible. In the same way that scientists for so long, before they discovered this so-called Higgs boson particle, pontificated that it was theoretically possible to discover such a particle, but the science was still out on whether it actually existed or not. They couldn't conclusively prove it, though everything said that it existed. With that same possibility, I can see it, it is possible that there are people out there not using Spring to build their services for software in production.
Josh Long (10:14):
So, for those hypothetical free radicals, Spring Boot is an opinionated take on the Java ecosystem. It's a framework that tries to give you the Ruby on Rails conventional configuration approach, while not taking any of the flexibility away. So it does that through not co-generation, which is notoriously hard to unwind once you've made that bet, but instead through this thing called autoconfiguration. And what that means is that when the application starts up, it looks at runtime queues, things like the presence or not the presence of a class in a classpath, things like properties in the environment, like an environment variable or system properties, and things like the presence or not the presence of other objects in the Spring application context, and it uses those to dynamically create objects that you need to support what you're doing.
Josh Long (11:12):
So, for example, if I wanted to create a web service, there are all sorts of things that you and I both know to go into that. There's a web server that needs to be stood up, there's conversion, there's actually a framework on top of the web server, so you've got your barebones HTTP service, but that's not the same as having a programming model on top of which you can expose endpoints, there's validation, the conversion, the routing, all that stuff that needs to get created before you, Jane or Joe Developer, can write their first line of business-differentiating value. And so Spring Boot says, well okay if we detect the Netty reactive web support on classpath or an embedded server container like JD or Tomcat or Undertow, it will automatically create a web server, and will also create Spring MDC, or will set up Spring WebFlux. It'll create everything so that by the time you start writing your first line of code, all you need to do is create one little class with an app controller on it, and now you've got a web endpoint.
Josh Long (12:10):
And this goes on. This opt-in mechanism where you express what you want in terms of the dependencies that are on the classpath, in terms of these other configuration queues, that autoconfiguration is very powerful and it pairs very nicely with this idea called a starter. A starter is a library, it's a dependency that is empty, it's devoid of Java code. Instead, it does only one thing, it transiently imports other dependencies. And so when you see Spring Boot Starter WebFlux, or Web, that brings in Tomcat and Spring MDC and the servlet API and all that stuff. It brings in everything I need for Spring Boot to autoconfigure all those bits required to create a web application. If I bring in Spring Boot Starter ...
Josh Long (12:56):
Is it Kafka? I think it's probably just Spring Boot Starter Kafka. It brings in everything I need to talk to a Kafka broker. And so we have starters all over the place. And you can remember those starters by memory like I sometimes am able to, but it's much more natural instead for developers today to go to start.spring.io, and there they can just shop a catalog of checkboxes, basically. You just go through them, there's a long list, there's auto-completion, you can discover them manually by exploring them, by browsing them, like strolling down the aisles of production. Or you can just type in K-A-F, and you'll get something interesting. Actually, you'll get a lot of different things that are interesting, in the case of Kafka.
Josh Long (13:35):
So, that generator there will generate a Maven build, and a skeletal source, main resources, source main Java structure for you. Or it will generate a Gradle build. And then that's it. The rest is all done, as I say, through this autoconfiguration mechanism. And the result is that your dependency management is trivial now, much easier than it would have been without Spring and Spring Boot, and the code is ... I'm now very happy to take my Java 15 code or my Kotlin code or my Groovy code and compare it to any Node.js program because we can get a lot more done with a lot less code these days. It's just a wonderful place to be. So I think that's ... the value proposition there is an opinionated approach to the Java or JVM ecosystem, and with a focus on production.
Tim Berglund (14:28):
By the way, I really like the way you say "start.spring.io."
Josh Long (14:34):
Oh, yeah. Say it again.
Tim Berglund (14:35):
You like saying that, don't you?
Josh Long (14:35):
I do.
Tim Berglund (14:35):
start.spring.io. You need to know, and I don't mind saying this publicly at all, that was an important inspiration for the naming of developer.confluent.io, the developer domain.
Josh Long (14:54):
Nice!
Tim Berglund (14:54):
Which is a thing that is very much still in the process of being built out. After we record this, I'm working on some content that goes there, some videos.
Josh Long (15:04):
Cool, I'll watch it.
Tim Berglund (15:05):
Videos, tutorials, executable tutorials, all kinds of cool things, for just learning how to use primarily Apache Kafka, and a certain amount of Confluent platform and Confluent Cloud, that stuff's allowed there since it's a Confluent platform.
Josh Long (15:19):
Right, of course.
Tim Berglund (15:19):
But hearing you say "start.spring.io," it's just so easy to say. Nobody walks out of a room, back when people were in rooms, nobody walks out of a room wondering "What was that domain? I can't quite remember." developer.confluent.io, it doesn't quite scan as well, because just the syllables don't work out, but it's super easy. It's better than confluent.io/developer. I just feel dumb saying that.
Josh Long (15:49):
Yeah, you should. I'm excited for you- [crosstalk 00:15:51]
Tim Berglund (15:55):
Right. Exactly. Just the name of it, really it's generational, but developer.confluent.io, it just works.
Josh Long (16:02):
It does. And I see the dots, I'm always happy when I see that. No, start.spring.io for me, as you may know, is my second favorite place on the internet. My first favorite place on the internet of course is production, I love production. You should love production. Everybody should love production. It's the happiest place on earth, it's better than Disneyland. But if they haven't been there, if they haven't already been to production, developers can begin their journey there at start.spring.io, so I hope they will.
Josh Long (16:31):
And it's a great way to bring in Kafka, it's a great way to build an app that isn't ... It's got everything, except for ... And actually if you've got a local host installed with Kafka, for your developer stuff, and you've got the normal settings, you don't need to do anything, you just start writing code to talk to Kafka. You inject the Kafka template and you can set up a stream list and everything. If you want to override it, point it to a different hosting port, a different cluster, a different whatever, you can do that, but the defaults are just [inaudible 00:16:55].
Tim Berglund (16:56):
Just to recap what you're saying, the annotation looks for stuff on the classpath, so you get the controller annotations. Which one, what do I do? Well, I found Jetty or I found whatever and it figures out the right way, it figures out the right thing to do and those things are present on the classpath, but the classpath by virtue of being in the built files dependency declarations, they're there by virtue of having been generated and IS checks the check boxes of start.spring.io.
Josh Long (17:33):
Yeah. Right. It's super cool. And it's opinionated ... So, Ruby on Rails is just a web framework, more about babysitting databases. You remember the use case, right?
Tim Berglund (17:39):
Oh, I do.
Josh Long (17:41):
But Spring Boot is, it's the conventional configuration for everything. If you want to do security, if you want to do batch posting, if you want to do integration, if you want to do BPMN, if you want to web services, if you want to do hypermedia, if you want to do [Rest 00:17:55], if you want to do microservices with Spring Cloud and if you want to do data flow, I mean, just everything, every little possible thing out there. If you can't find the starters on start.spring.io, that doesn't mean that they don't exist, that just means they don't meet our fairly high watermark for inclusion on start.spring.io. So, we don't include every single random check box in the ecosystem, only those that have a vibrant community that have committed to maintaining those starters with each release of Spring Boot, so with each new release.
Josh Long (18:26):
So, you'll find the check boxes for Ali Baba, for Google Cloud, for AWS, for Microsoft Azure. Those are there because they have people that work on that every single time there's a new release of Spring and Spring Boot. So, you never have interoperability issues, it's never the case of one check box on start.spring.io requires an older version of Spring Boot than the one that you selected on start.spring.io. So, it's a really high bar, but as result, you can just ... it just works. You can just write code and not worry about it. But that's not to say that others don't actually maintain their own starters, right, it may be the case that they do. You can check your favorite projects, they might even have their own starters.
Tim Berglund (19:08):
Exactly. And that's just, again that's a Maven dependency for a new build, right?
Josh Long (19:15):
Yeah. And it's a big deal, right, it's this ... You and I are ... Or just me, okay, I'm going to say me.
Tim Berglund (19:22):
Just you?
Josh Long (19:24):
I like just the software, I really don't care about Gradle or Maven, not even a little bit, not one bit, zero percent. It's not a hill I'm willing to die on or even ... it's not even a hill I'm willing to get on top of [inaudible 00:19:36].
Tim Berglund (19:38):
The hill isn't even on your maps, your troop movements, you're just ... Right.
Josh Long (19:43):
Yeah. Some people choose Google Maps and they choose toll roads and stuff like that. Me? I just choose no hills, none, I just avoid them at all costs.
Tim Berglund (19:51):
Yeah. Not going to do it.
Josh Long (19:53):
Yeah. It's not even on my radar, I don't care, it means zero, it means nothing to me, it means so little to me that it's actually ... it will actually cause me to yawn, just talking about it. Right? So, the idea that people get all bent out of shape over their build systems is ... What am I supposed to do, right? You might have to just leave the room. Is that right? Is that polite? I don't know. What's the right answer there? So, Spring Boot doesn't carry you there, but we want to make it so that it's as painless as possible to get started, so you don't even have to focus on it, you can just ... You opt in one dependency that equals the whole tranche of LED functionality, one dependency that equals the core support from Kafka. And that is, by the way, interesting, because the core support from Kafka is just that, it's just the foundational layer. We have layers here, Tim, layers. Okay?
Tim Berglund (20:34):
Okay.
Josh Long (20:35):
So, in the Spring ecosystem-
Tim Berglund (20:37):
Like onions? Like onions or like cakes, which ... more like cakes?
Josh Long (20:40):
Actually, I'm ... My partner, Tammy, she loves onions, and she might disagree. In the Spring ecosystem we have the core Spring [inaudible 00:20:52] Kafka projects, and that project gives you ... it will give you the basics, it will give you the ability to connect to a Kafka instance and you can actually get a reference to a Kafka template, so you can use that to fire off messages with one-liner, that kind of thing. You can set up a message list or a container, like the old JMS, message driven EJBs, right, the idea of a container that receives messages. So, all that is in the core of Spring for Apache Kafka starters. So, you go to start.spring.io and you choose that one. You're already talking to Kafka and you're already sending and receiving messages.
Josh Long (21:31):
And you have a low enough level of an abstraction that you can do anything that you want to do, you can actually get access to the low-level Kafka APIs and so on there, which is already valuable. If you're trying to do something very particular, then that's fine. If you're trying to do something just very generic, there are also suitably high-level attractions implied in that dependency to support that use case as well. But we can further, so we have a project called Spring Integration, Spring Integration's pipes and filters, messaging framework. Right? It's a framework for building enterprise application integrations in the style of Gregor Hohpe's and Bobby Wolfe's seminal tome Enterprise Integration Patterns. Right?
Tim Berglund (22:20):
Yes.
Josh Long (22:20):
That page-turner of like 800 pages from 2004, that I keep under my pillow.
Tim Berglund (22:25):
Yes. Yes.
Josh Long (22:27):
That framework is, it's all about integration. Right? And as you and I both know, you can simulate something that is synchronous with asynchronous API. If I have an asynchronous API, I can force it to act from the perspective of the client as an asynchronous API, but I can't do it the other way around. I can't take asynchronous API and emulate an asynchronous one. Right?
Tim Berglund (22:53):
Yeah. Well, it's difficult.
Josh Long (22:57):
Right. So, basically from an innovation perspective, it's better just to assume that everything is asynchronous, and that way you can code a model if you write the code in such a way that everything is asynchronous. Then, if you want to, you can have everything executed on the same task executor, the same thread pool basically, and then it's effectively synchronous, but the API is still asynchronous. So, you have this pipes and filters model where messages come into one component, they get processed in some way, then they get sent out. And so the contract between the components is just like a standard in, message out, the message in, message out, message in, they're chained together. So Spring Integration, in order for it to do its work, it needs to source a message from something in the real world, something out there far beyond the sandstone-
Tim Berglund (23:40):
The boundaries of ... the firm foundation of the framework.
Josh Long (23:46):
Right, yeah. Far beyond the sandstone beaches of the comfort of a framework in abstraction. Right? So, something like Kafka, something like FTP message mailbox or a FTP server or a TCP NAP package or MTT framework, whatever, anything in the world that isn't the soft cushy cozy confines of my Java compiler. Right? I want to take that event out there in the world and turn it into a message that I can deal with in [Java 00:24:14]. Right?
Tim Berglund (24:14):
Right.
Josh Long (24:15):
So, Spring Integration uses an adapter to do that. Again, this is terminology borrowing from that book. And these adapters allow us to take messages in and we're done processing it, we can take that message and send it out to an outbound adapter. So, inbound adapter, outbound adapter. Sometimes when we send a message out, we want a response, so it's a request a reply flow instead of just sitting there with a file and forget flow, so that's called an outbound gateway. And also the other way around, when I have a message that comes in and I'm expected to produce a reply, that's called an inbound gateway. So, Spring Integration provides Kafka support, there's a module for Spring Integration in Kafka, so you can do that. Right?
Tim Berglund (24:52):
Mm-hmm (affirmative).
Josh Long (24:53):
There's also Spring Cloud Stream. And Spring Cloud Stream is an abstraction for building ... it's an abstraction that says, "Look you want to connect, you want to get a message from A to B, right, you want to just ... you want to send a message to this broker or to this thing over here," but I don't want to write code in terms of the low level Kafka APIs and low level RabbitMQ APIs and low level Kestrel and Google Cloud, Pub/Sub, whatever, I just want to write code in terms of the Spring Framework MessageChannel, so I can say, "Okay, here's the MessageChannel, send a message into the MessageChannel," and by virtue of the external configuration, that message will get delivered to something else. I bind that Java interface reference that I've got to an actual exchange or destination in a broker somewhere.
Josh Long (25:42):
And there's another module in Spring Cloud Stream that has a similar function, but it doesn't work in terms of the MessageChannel, it's actually Spring Cloud Stream for Kafka Streams. And this is my favorite, right, so Kafka Streams is super, super cool, Kafka Streams is like ... I don't ... I can just like ... half the use cases for which we might have used Spark yesterday, suddenly I can do on my little Spring Boot app. Right?
Tim Berglund (26:08):
Exactly.
Josh Long (26:09):
Which is to me just a winner, it's a huge deal. But nonetheless, I'd like things like conversion, like message conversion and so on to be a little bit easier. I'd like things like actually constructing the instances of the K-topics and the K- [crosstalk 00:26:25].
Tim Berglund (26:24):
KTables and KStreams.
Josh Long (26:27):
Yeah. KStreams and Ktables. Yeah. I'd like the sourcing of those objects to be a little bit easier. So, Spring Cloud Stream for Kafka Streams does a lot of that. So, it's really ... you're still dealing, in terms of these very nice integrators, as opposed to the glue code you're so often having to worry about otherwise. Right?
Tim Berglund (26:44):
Yes.
Josh Long (26:44):
So, it's a very nice mix. And that's one of my favorite products in the Spring ecosystem actually, Spring Cloud Stream for Kafka Streams, because it gives you ... you can treat Kafka like a regular message queue if you want, but-
Tim Berglund (26:58):
You shouldn't, but if you had to.
Josh Long (27:01):
Right, yeah. Yeah, that's right, it's if you want. Right? But the Kafka Streams thing is like, "Ah, that's ... now I see why people are really, really enamored with Kafka." Right?
Tim Berglund (27:13):
Right.
Josh Long (27:14):
It alleviates or it obviates the need for just whole categories of other infrastructure, other deployments.
Tim Berglund (27:24):
Other computation and storage needs that you have. The boundaries of the platform go well outside of a pipe between components.
Josh Long (27:31):
Exactly. Yes exactly. And really, just brilliant, I'm just so enamored with that. You know, I've been doing a lot of ... Here's the thing, I've been doing a lot of reactive stuff, like the last two or three years.
Tim Berglund (27:46):
Relevant to our interests?
Josh Long (27:48):
Right. And so you start thinking about data flow and composition in terms of these functional APIs. Right?
Tim Berglund (27:56):
Mm-hmm (affirmative).
Josh Long (27:57):
So, then if found, oh okay, wait there's this Kafka Streams thing that ... it's not exactly reactive, but it's very much the same idea, it's this pipes and filters architecture, you have these functions that operate on streams of data, you're just building this pipeline. And it just maps naturally, just one-to-one, to what I'm thinking it'll do. Absolutely amazing, what a privilege for me to be able to use that.
Tim Berglund (28:20):
And there are a few things you said that I want to talk about. You did say something like this keeps you from having to construct this that and the other object just to get simple work done which is ...
Josh Long (28:34):
Uh-huh (affirmative). Uh-huh (affirmative).
Tim Berglund (28:36):
There's 15 years' of a core Spring value right there. Spring has always been trying to keep you from having to do that.
Josh Long (28:44):
In-version control, somebody can do it, just not you.
Tim Berglund (28:47):
Right. And that's your passion. You said at the beginning what you care about is the code and you don't even want to talk about the build. Well, it's good, somebody gets to care about the build, but that's just consistent with I think what you do, teaching people about Spring. Let's focus on the differentiating value that you're trying to create and get all the other stuff out of the way.
Josh Long (29:11):
You know, we have people in the Spring team that have no hair, they're completely bald, no hair at all. That's because they spent time worrying about these dependencies so that we don't have to. Don't dishonor their sacrifice by then plunging the depths of the build magnitude, it's just such a waste.
Tim Berglund (29:31):
Don't do it, don't dishonor their sacrifice, their work has not been easy.
Josh Long (29:36):
No, it's been a nightmare.
Tim Berglund (29:37):
And we just don't want to profane it by worrying about these things that we don't have to worry about if we're using Spring.
Josh Long (29:45):
Yeah. I mean, I'm not saying there's not going to be any dependency that we haven't thought of that you might have to integrate, but even then hopefully if you bring your dependency and it has a version of lock for [Jay lever 00:29:59], by bringing it into the Spring Boot build we've already level set the version of lock for [Jay 00:30:03], so hopefully you get consistency there.
Tim Berglund (30:06):
That's worth ... we're talking about open source software here, but that alone is worth hundreds and thousands of dollars to plenty of teams, just getting logging out of the way. If you're not a Java developer, then you just don't know, you don't. And it's not something Josh and I can talk about in public really, it's a painful thing. If you haven't experienced it, you don't know.
Josh Long (30:28):
Oh yeah. Well, you and I came from a time when first of all the build was a nightmare before, it was even worse with [Ant 00:30:36], now we have the tools to level set stuff, but with Ant it was just like, "Does this work? No. Okay, throw that out. Does it work now?" And then there's also the runtime configuration, the reams and reams of deployment descriptors and all this stuff that by the time you've finally gotten to Hello World, you're just tired, you're just ... I don't even care about the world anymore, I'm just going to leave it here. The five-minute [MMO 00:30:59] was not possible.
Tim Berglund (30:59):
Can't even say hello, yeah.
Josh Long (31:01):
Yeah, just hi. So yeah, now we're deployment descriptor free, we're mostly building management configuration free and we can focus on these higher order atoms, these higher order bits, these higher order things that matter to my business, which is, again, production. You know what, that's a good point actually, which is take 2020, if you ask me or arguably ECTO, and I'm sure you if you ask anybody who's been around for a minute about what they want most in an engineer, they don't want that ... nobody wants a lone cowboy or cowgirl, nobody wants this narcissist, nobody wants that, nobody wants a person that insists of having ownership of one section of the coding.
Josh Long (31:59):
What they want is people who are focused on production, people who shift. And so I just think it is so nice that today the things that ... Because you remember when we started doing Java and there was this idea that what you did in your development machine, which by definition was not going to be the same thing as your production machine, needed to be reconfigured before it got to production. So, you would develop on Windows, because what else would you develop on, right?
Tim Berglund (32:23):
Sure.
Josh Long (32:25):
And then you throw it over the wall and some people ... lots of people would configure it and change things and then it would be run in a Solaris machine, and it's real quick. That's not the same environment at all. And that took a lot of regression testing and it took a lot of like ... Yes, Java gave you the right to run everywhere, but everything else didn't- [crosstalk 00:32:44] What?
Tim Berglund (32:46):
I was kind of hoping you were going to use those words. It sounded like that's where you were going, right?
Josh Long (32:50):
Yeah.
Tim Berglund (32:50):
But the reality of the tooling, the frameworks we used back then, it was written once and then run very carefully in exactly one other place with all of this bespoke baling wire and duct tape all around.
Josh Long (33:09):
It was crazy, it just absolutely boggles the mind. And of course, now it's not Solaris, I mean, I love ... a call out for Solaris, I love Solaris, but it didn't win, it's not this [uber de jour 00:33:19], and nor is Windows, right, it's Linux. So, that's not hard, I can make sure that I'm running on Linux when I develop or reset a virtual machine or [inaudible 00:33:27]. And then database isn't ... I'm not using an embedded database in my development environment and then using some overly expensive thing that I can't run also in my development environment, there's no ... It's no longer the case that I need to have production only licenses and that developers need to pay $100,000 or whatever for their own developer license or whatever, some crazy thing like that. If I'm using [POS King 00:33:50], I can use it in both places.
Josh Long (33:51):
If I'm using Kafka, I can use the same thing in both places. So, now the gap between production and development is gone and so the next thing is if all that is gone and we don't have to maintain these two profiles for the code base, these two branches, development versus production. If all that's gone and now I can focus on higher order stuff, what else is possible then? I mean, we're not forced to think about this low level stuff. Well, for me it's like, "Okay, of course, I can start thinking about building more reliable services." Now that we've got this cloud architecture, we're so far removed from the individual instances of the service that we're deploying, we're so far away removed from the infrastructure.
Josh Long (34:34):
And our code is so far above all that, that now we're thinking about these really like, "Okay, if I send a message to this service, what happens?" And it's not hard to get a service-connected to another via Kafka, it's easy. I just think that focus on production, that focus on high order abstractions, is super valuable and I wish we had it 20 years ago.
Tim Berglund (34:58):
You know, that's the thing, it would have been better, but I don't know ... This is the keynote I gave at the Kafka Summit, I developed this idea in a little bit of detail and I'm giving it at some more events this fall. There's always a developmental process that we go through, and this happens in technology stacks, in communities, people in a relationship, children do it, it's everywhere, it's just living things develop. And it would have been nice if we'd had that, I don't know if we would have known what to do with it as we want at a point where it made sense. Looking back, it's like, "Wow, I wish I had all those tools then," but we had to grow them.
Josh Long (35:38):
Yeah. But I think people were struggling with deployment, we've always had this ... I don't think anybody ever said, "Oh, what do you mean?" I mean, there were some people I'm sure, but they were vendors, I don't think many people were very upset when we finally got the tool to say, "Oh, I no longer need to care what operating system I'm deploying."
Tim Berglund (35:56):
Exactly. I know. And it made all the sense in the world. And, I mean, I'm sure there're things that would have been helpful. But for [inaudible 00:36:05], but the way we talked about the cloud when the cloud was new and I would try to explain I've read this article on this thing called S3 and it's like a big disc in the sky. "Okay, that's easy." No, there's this thing called EC2 now, as you can just make servers, and so at Christmas, you can just spin out more servers for your retail business, and then in January, you spin them down so you don't pay for them. I wasn't building applications that could do that then, nobody was, it was a true statement, that's what the service did, but we weren't really developmentally capable of using it in that way, because the rest of our ecosystem had not grown ... I mean, Spring Boot happens to be an interesting part of that specific development process, being a framework that lets you build microservices.
Josh Long (36:57):
Right. Oh, you're talking about the microservices, yes, yes, yes, absolutely. Microservices ... But you're right, we still built Monoliths, when the cloud-native era arrived most of us were building Monoliths. You're right, but on the other hand ... And again, I work at VMware, but I'll be the first to say it, I don't know much about Vms. One of the VMware use cases, one of the original use cases was, "Can I develop against a virtual machine on my local machine? And how to view a copy of production, or better yet, can I standup infrastructure that I thought was only in production or some shared environment that I needed to be on a VPN for. Can I just get that on a local machine and take it home with me, so I can develop locally?" [crosstalk 00:37:45]
Tim Berglund (37:47):
... all kinds of things.
Josh Long (37:49):
Crazy. Yeah, slow, yeah exactly. So, that was phenomenal to me, that was a game-changer, to say, "I can just package production and just use it, so I can try it out without hurting production. Wow." And so when we go to AMIs, the Amazon and virtual machines and all of that, well, of course, we could take that packaged production and just clone it, and just have more of them in production. Right? Why couldn't I, that makes a lot of sense actually? And then you get containers and it's just ... You're right, we did have to grow into it, but it just seemed so natural every step of the way. It forced a sea-tide change in the way we built software, and so now you start thinking about what happens if you have lots of services. Right? Well, of course, you want to communicate.
Josh Long (38:32):
Well, suddenly RPC doesn't seem very reliable, that's not a pattern I think any of us want to espouse, when we know better. So, send me messaging, send me cloud-scale messaging, it's just obvious, obvious, that something like Kafka was going to just crush it, just absolutely kill that whole use case, just make it like ... Well, of course that's done for you, it's done, just the whole production, you're done, don't worry. It's like if it serves how many billions of requests per whatever at Netflix, then it will probably do just fine for you too. It's so crazy.
Tim Berglund (39:09):
Another question I've had bouncing around in my head is, you've been talking about applying the sensibilities of the Spring Framework to messaging, which Kafka is unlike legacy messaging systems in important ways. But you said something more important, like you can bind the message to a particular destination, which is a messagy sort of way to look at it. And I know that "destination" can be a topic and so it's fine that Kafka adds that abstraction. Which leads me to the question, something you have talked about in person before, this is a general Spring question.
Tim Berglund (39:58):
You have all of the APIs in all of the world for the things that you want to do and 90% of the API surface area is noise to you as a developer, there's really this kind of parade of distribution of features and frameworks. This is the little chunky one, the rest of it is [inaudible 00:40:17] and Spring makes that chunk all work in a consistent and Spring-like way, so you learn ... Is that literally one API, one accent, and speaking that dialect and it's easy and you do that to Kafka. But what am I going to say? I'm going to say when you wrap APIs, the abstraction's huge because sometimes you have to break out into that other 80%.
Josh Long (40:42):
Uh-huh (affirmative).
Tim Berglund (40:44):
And so ... I mean, this is me 100% endorsing the approach of building ... Just the normal story I hear of people actually using Spring Boot, is they're building microservices and if Kafka's in the picture, then their microservices are integrated to Kafka and it's a beautiful world, it's like living in a Star Trek episode.
Josh Long (41:04):
Yeah. Better.
Tim Berglund (41:05):
What's that? Even better?
Josh Long (41:06):
Better.
Tim Berglund (41:08):
Yeah, with no Borg, with no anything.
Josh Long (41:10):
Yeah exactly. Well, actually- [crosstalk 00:41:13]
Tim Berglund (41:14):
Yeah. But yes, that's like all the right stuff and Spring Boot is a part of that. But what if, what happens when, how do those abstractions meet when you really need to just be in the Kafka mode of things and not the Spring mode?
Josh Long (41:29):
That's why you'll see us introducing it in terms of layers, right. I alluded to some of that earlier, I talked about autoconfiguration. Let's start with low level stuff, like autoconfiguration. The idea behind inversion of control or dependency injection is that something central is figuring out the wiring of all your objects. And you can write code say, "I have expressed a dependency on this type." Well, I'm expressing a dependency on, let's say, a data source or a connection pack or [inaudible 00:41:56]. I don't care how that thing was realized, I don't care how it was ... resource initialization and acquisition, that's left to the container, that's left to Spring. So that's part one. So, my surface area there isn't a particular API, it's just a ... it's an interface, that's a generic interface, it has nothing to do with Spring. So that's the first bit, you decouple them that way. But you can also inject specific types. Right? And the other thing is, because Spring is doing all this stuff, there are lots of different objects that are being created for you.
Josh Long (42:26):
So, like I mentioned in the example of the web server earlier, what if you wanted to change the way that the embedded Tomcat was created? Well, in this case, you just register a [beam 00:42:34] of the same type as the one that we're registering, and Spring Boot knows to back off. Anywhere it sees that you've expressed a more specific opinion than the one that its got by default, it short circuits the advantage of the one that it provides out of the box and it defers to yours. So, that's one way where you can have the defaults, the common conventional configuration defaults, but override the defaults, if you like, at any given juncture. And this is true for any of the thousands of objects that get wired up for you automatically in all the different starters in all the different wonderful wild world of Spring Boot. That's the first part.
Josh Long (43:05):
The second part is, when you get to these abstractions, the actual concrete types against which you must compile, now you have layers. Right? So, we talked about the Kafka template, well the template in the Spring ecosystem is a client that gives you APIs that map to common one-liners. Like the Kafka template has something like convert and send, so you give it regular objects in a string maybe and it will send a message, it will take that object pass it through a convertor and then send it to the destination implied by that string, something like that. Right? Well, that's very high level and it assumes a whole bunch of things. Well okay, there's also a version that also takes, instead of just a regular Java object, it takes a message, which you expect it to have already ... It's in an envelope, it's a header in a [pay lode 00:43:51] and you expect it to already have converted that [pay lod 00:43:53], for example, then you provide headers. Okay, that's a little bit more conventional, but it's still very high level.
Josh Long (43:58):
Well, okay, there's another version that probably takes something like a callback where you get to execute code within a session, you get access to a current client, and you can do everything you would at the lowest level of the API, but you're not worrying about how to acquire the client or anything else. You're just given a callback that says, "This client is ready for you, here it is," When you answer the callback, then it gets send. Right? So, the same template will have all these different layers. That's one way. Right? I mentioned Spring Cloud Stream, which has this very simple message channel in, message channel out kind of modality, and I said that's very ... that does sort of trivialize what it is to have a message broker, to have a message queue. It is really just a pipe, message into the pipe, message out of the pipe, message into the pipe, and out of the pipe.
Josh Long (44:48):
And you're right, you sacrifice some of the flexibility there, but for this common case of just send data there please, that works well. That said, we also have Spring Cloud Stream for Kafka Streams, and there you're given the rich lower level ... given the actual low level types of KStreams and KTables, with which you can stuff, so you're not giving up on that either. So, I think it's just really opinion, and you can mix and match, using Spring Cloud Stream for Kafka Streams doesn't preclude from also then using this Kafka template at the lower level, sort of just get access to the lower level API and do something with it. Right?
Josh Long (45:21):
I think that's the right answer, I think we just put out layers and we say, "If you want to work at a high level, a very high level, then be our guest, start here, this will get you sending data in and out. But if you want to also then get down into the Kafka Streams bits, you can use this particular thing over here. And if you want to get down to the low level Kafka APIs, down to the lowest level, then you can use this template or just inject the client itself, you can actually just ..." You get access to the objects that are required by the Kafka API, the types have [inaudible 00:45:52], Kafka or whatever, in the types, in the packages.
Tim Berglund (45:55):
Yeah, which is ... I mean, that's the thing you want. If you've got a layered system, you want to be able to peel off the layer that you're in and directly access the layer underneath.
Josh Long (46:08):
Right.
Tim Berglund (46:08):
And as a developer using the layer of abstraction, you do want to go one layer down. Not because you want to touch it, because remember the parade of distribution features in the API, you probably don't need 80% of them and the 20% of them that you do need, you've got this nice abstraction that adds its own value and you're happy using it. But it never works all the way, you always, always have to go that way around, so you know those types underneath there, that API underneath there, you can get access to them, and that's a good thing.
Josh Long (46:40):
Right. Yeah, you're absolutely right, I should have started with that, which is everything you can do in Kafka you can still do via virtue of the interaction that Spring provides. But if you want to just ... if you want to step up the ladder a little bit, you can, and even then ... Actually Kafka Streams, I just mentioned, it feels very rude to say that, I just say, "You can use the low level Kafka Streams." Kafka Streams is ... To anybody from the old messaging world, Kafka Streams feels like ... it's in the sky- [crosstalk 00:47:11]
Tim Berglund (47:14):
Right, it's not a normal API. But, and I can personally attest from our work together with Spring Cloud Streams, it was a year and a half ago and it was fairly early and we needed some help getting it running in a few ways. But you really had that system where there was the friendliness with the rest of the Spring ecosystem, but then you just ... you got KTables and KStreams and you got all that APIs. It's not like Spring has attempted to create its own generalization of that with Beam or something, it's a very singular approach, in my mind, I like it a lot.
Josh Long (47:51):
All impact for the Kafka Streams stuff what I found was you and I took code that was just made of naked, other than it being Kafka Streams code, and we just ... and we got it working inside of Spring and then we just started deleting things. It was the same code, we just kept ... we were able to just remove certain parts of it, that's all. It's the same exact code, we didn't add new code, we didn't have to change it, we just removed. You're still using Kafka Streams, which is awesome, because Kafka Streams ... number one, awesome, I'm a big fan, as you know.
Josh Long (48:21):
That's, for me, truly, truly, truly that was when I started becoming ... when finally it clicked for me. I'm a simple person with simple ideas, so it took a long time for me to really appreciate it, "Oh, oh that's why people keep talking about Kafka as though it's this ..." It's not just a pipe, it's not just a faster RabbitMQ or JMS or whatever, it's the completely wrong way to think about it, that's not the way to think about it. You're missing the fact that there's actually a database there, and that database can do State computation. Amazing.
Tim Berglund (49:01):
My guest today has been Josh Long. Josh, thanks for being a part of Streaming Audio.
Josh Long (49:05):
Thank you, Tim, for having me, it's been a pleasure as always, my friend.
Tim Berglund (49:08):
Hey, you know what you get for listening to the end, some free Confluent Cloud. Use the promo code 60PDCAST. That's 6-0-P-D-C-A-S-T to get an additional $60 of free Confluent Cloud usage. Be sure to activate it by December 31st, 2021 and use it within 90 days after activation. And any unused promo value on the expiration date will be forfeit and there are limited number of codes available, so don't miss out. Anyway, as always, I hope this podcast was helpful to you. If you want to discuss it or ask a question, you can always reach out to me @tlberglund on Twitter. That's T-L-B E-R-G-L-U-N-D. Or you can leave a comment on a YouTube video or reach out in our community Slack. There's a Slack sign-up link in the show notes if you'd like to join. And while you're at it, please subscribe to our YouTube channel and to this podcast, where ever fine podcasts are sold. And if you subscribed through Apple podcasts, be sure to leave us a review there. That helps other people discover us, which we think is a good thing. So, thanks for your support and we'll see you next time.
Tim Berglund (50:15):
[music]