Confluent Developer ft. Tim Berglund, Adi Polak & Viktor Gamov

Real-Time Payments with Clojure and Apache Kafka ft. Bobby Calderwood

• Confluent, original creators of Apache Kafka® • Season 1 • Episode 70

Share episode

Use Left/Right to seek, Home/End to jump to start or end. Hold shift to jump forward or backward.

0:00 | 58:00

Streamlining banking technology to help smaller banks and credit unions thrive among financial giants is top of mind for Bobby Calderwood (Founder, Evident Systems), who started out in programming, transitioned to banking, and recently launched Evident Real-Time Payments. 

Payments leverages Confluent Cloud to help banks of all sizes transform to real-time banking services from traditionally batch-oriented, bankers’ hours operational mode. This is achieved through Apache Kafka® and the Kafka Streams and Kafka Connect APIs with Clojure using functional programming paradigms like transducers. 

Bobby also shares about his efforts to help financial services companies build their next-generation platforms on top of streaming events, including interesting use cases, addressing hard problems that come up in payments, and identifying solutions that make event streaming technology easy to use within established banking structures. 

EPISODE LINKS

SEASON 2
Hosted by Tim Berglund, Adi Polak and Viktor Gamov
Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed
Music by Coastal Kites 
Artwork by Phil Vo 

  •  🎧 Subscribe to Confluent Developer wherever you listen to podcasts. 
  • ▶️ Subscribe on YouTube, and hit the 🔔 to catch new episodes.
  • 👍 If you enjoyed this, please leave us a rating. 
  • 🎧 Confluent also has a podcast for tech leaders: "Life Is But A Stream" hosted by our friend, Joseph Morais.
SPEAKER_01

Now, even if you don't know a lot about banking, you probably have a sense that banking's computer systems run a lot of batch jobs. The good news is that's changing, and Bobby Calderwood's company, Evident Real-Time Payments, is trying to help. I talked to Bobby today about payment systems, Kafka functional programming, even a tiny bit of closure. It's all on today's episode of Streaming Audio, a podcast about Kafka, Confluent, and the cloud. Hello and welcome back to another episode of Streaming Audio. I am, as ever, your host, Tim Berglund, and I'm joined today in the virtual studio by Bobby Calderwood. Bobby, welcome to Streaming Audio. Thanks, Tim. Happy to be on the show. Awesome. You've been a guest blogger for Confluent in the past. Um, I remember that was back in the days when I still personally edited every blog post. And I remember reading yours, and I think a single tear escaped my eye as I read it. Because it was just like I I think everybody listening to you needs to know. It was solidly written. It just was like I didn't need to really give it a whole bunch of love from a copy perspective. And the technical content was awesome too. So thank you for that. You can't hear me, but I'm blushing. See, right, right. Uh little moment of joy in my in my blog uh post copy editing work. It was was wonderful. Um anyway, we're not here to talk about you as a blogger, but that's an important part of your introduction, and I I will definitely have that link in the show notes. Uh, but I would love to hear a little bit about you, and I think everybody would. What um what do you do? What's your background? Tell us about you.

SPEAKER_00

Yeah, absolutely. So um when I wrote that blog, uh I was actually at Capital One and I was uh distinguished engineer there in the Tech Fellows program, and we were just trying to tackle a lot of really hard problems in financial services. Uh and one of the initiatives that I was trying to spearhead was the use of streaming data, use of event sourcing, uh the use of CKRS, and you know, building systems out of uh logs of business events. And Kafka is just an amazing uh enabling technology for that uh that category of applications and services. So uh that was a really interesting um time and and really enjoyed the work that I did there. Uh before going to Capital One, um, I worked at some startups. And and prior to that, I worked at uh Cognitect. Uh that's the the company that stewards the closure programming language. Um I was working on a database called Datomic. Uh I was on the product team there for a while.

SPEAKER_01

You said you said stewards. I thought you said stewards.

SPEAKER_00

Uh there are there are several stewards there, several notable stewards. Yeah. It's the company that also stewards uh closure.

SPEAKER_01

And yeah, okay. And what it was probably called relevance back then by that point in the time.

SPEAKER_00

Yeah, when I joined, it was relevance, and I was doing mostly um kind of the professional services side of things, built some really cool systems for for some neat customers uh with relevance. Um Rich Hickey uh uh merged the the the company with relevance, uh his company, um, and it became Cognitect. And then I joined the uh the product side of the house with Rich Hickey and Stu Holloway and uh David Nolan and Chris Redinger and Timmy Wald, but a bunch of really, really great engineers. It was it was that was like the dream team. I don't I don't know if I'll ever be on an engineering team that amazing uh again in my career.

SPEAKER_01

But it it like I we should just link to everybody's Twitter bio in the channel. You really should. It is that is an absolutely amazing uh group of engineers there. And uh a few of them I'm I'm privileged to know, and they really are brilliant people.

SPEAKER_00

So yeah, they're great. And I was definitely I was definitely Ringo Star. Uh I was I was sort of the the junior junior member at the least talented engineer, but I learned a ton. It was a great education for me.

SPEAKER_01

So Bobby, I'm gonna step down. I uh occasionally uh have socialized with like some set of the some subset of that group, never even written code by them. So your Ringo car, Ringo Star, I'm Pete Best. Uh that's that's but you know, hey, I'll take it, right? That's right. Um cool. Okay, so that you are uh closure and functional programming and stream processing and event sourcing and CQRS person of fairly deep experience there.

SPEAKER_00

Uh yeah, and a lot of that kind of event sourcing and CQRS came from uh from Datomic, you know, being in that system, helping customers use it, um sort of really imbibing that architecture uh that Rich put together and that Stu helped uh to really implement and make real. It was it was inspiring and really got my kind of wheels turning on this whole CQRS event sourcing architecture. Now Datomic's down kind of in the in the data domain, using those techniques in the data domain, um, but I envisioned it uh, especially as I went over to Capital One, more up in the business event domain, which is, you know, there's some differences, some nuances there in terms of granularity and level of abstraction and so forth. But uh yeah, a lot of a lot of that work that I've done since leaving Cognitect, I definitely owe to that architecture and seeing it and and working with it and imbibing the the ideas there.

SPEAKER_01

Yeah. And I I remember uh when I first saw your blog post, I was still like relatively new to the space. Uh it was early on in my tenure at Confluent and you know in my participation in the Kafka community broadly, and there was a real light that went on where it was so obvious that all of those functional programming concepts that are um you know uh absolutely in the blood of the closure community. If you've written some closure, if you've interacted with that community a little bit, uh like you know, nowhere except maybe Haskell and Erlang, do you see people more resolutely committed to these principles and and sort of living them out in their code uh more um more directly. And then all of those just applied perfectly to Kafka and the kinds of systems that Kafka wants you to build. Uh that was what you did. That was you know what you accomplished in that blog post was uh showing to me it was a kind of a should have had a V8 moment where you know it all seemed so obvious that yes, of course, this is functional programming writ large uh in in kind of at a system level, at an architecture level.

SPEAKER_00

Yeah, yeah. Yeah, that that was uh you know, both uh definitely, you know, bringing some of those techniques that that are are like you said, kind of in the DNA of the closure community and in the design of Clojure and how you use it and the types of systems Clojure want you to write. Um and it was also a reaction against um some of the kind of micro systems uh blunders and excesses that I saw in my, you know, in my work with with a lot of different companies. But you know, seeing people yeah, you know, seeing people do the the kind of distributed object thing again. Like, I don't know, we've just been down that road in the technology world a couple of times, and it's just not the right thing.

SPEAKER_01

Yeah, it's excessively more traumatic each time, but yeah, that's right. That's right. Keep trying. Uh okay, all right, that's good. I didn't I didn't pick that up, but I'll have to go back and revisit it and see if I can see that. Uh shortly after that time, though, that was a couple of years ago, you left Capital One and started a company.

SPEAKER_00

Yeah, that's right. Uh so I left Capital One, started a company called Evident Systems, um, with the goal to bring to banks, especially smaller banks and credit unions, um, the type of system we've been describing, bringing them systems that are evident, that give control back to the system user, that show the user what it's doing and why, and you know, less black boxy systems and more evident systems, systems that you know are self-describing and uh uh easy to monitor, easy to operate, easy to you know, comprehend and reason about. So trying to bring those types of systems to financial services, the state of banking technology uh in this country is is like really bad. And that was another thing I learned at Capital One is it's sort of like you know, banking uh computerized really early on. And so it's been dragging along this this kind of long chain of legacy technology. I mean, no kidding. Yeah. At Capital One, I was like dealing with mainframes and you know exporting data out of them. It was just it was bonkers.

SPEAKER_01

So trying to uh if I if I can't riff on that for a second, airlines too, anything that was was at a sufficient scale in 1960 to to want to you know to have money to buy a mainframe seems to have like this terribleness to its own.

SPEAKER_00

Healthcare too. Yeah, it's like airlines, healthcare, um, some heavy industry, and and finance. It's just brutal.

SPEAKER_01

But I've and and I don't want to well, no, maybe I do want to rabbit trail in this. Uh I'm it's I'm interviewing you. Uh but the but the the uh payment technology in the United States. So I've noticed, for example, I travel a lot, and you know, 10 years ago in Europe, uh everything was chip and pin. And I had this like primitive card uh that you had to swipe and get a signature, and there'd be some places, like in Scandinavia, you know, some young person who's new at the job, they they wouldn't actually know how to use the card. Yeah, you'd have to get the old machine out of the back room to bring your action. Like in Oslo, all the restaurants use the same terminal. It's everybody using anything. Oh, you press the press the yellow button, press press the yellow button, okay. Now swipe it. Uh and then, you know, so that's embarrassing. And then we get chips, but they're chip and signature, they're not chip and pin. And now in Europe everything is kind of becoming contactless, and I don't even so what I I find it difficult to believe that um they computerized early is the explanation for that. Do you have any insight into why that seems to lag here?

SPEAKER_00

Yeah, so um computerizing early definitely is part of it, not so much from a direct payment technology perspective, but from the um the integrations that you have to do between your payment technology and your uh account level, you know, account management system of record, the thing that like maintains the balances and records all the transactions and so forth for a given account. Um, a lot of those systems uh truly are to this day still mainframe systems or even kind of older mid-range systems. But uh the payment technology is sort of the um the best part of the stack at this point in the in the banking world. You know, like I said, a lot of these kind of general ledges general ledger systems and um uh account management systems like systems of record that that maintain the transactions and account balances and so forth, those are really the stodgy old things, but everything has to integrate with them, and so it makes everything go slow. So some of the drag we see relative to again, like Europe or something like that in the credit card space or in the payment space definitely does uh come back to this. The other real difference between or or I guess the idiosyncrasy in the US banking space is just the scale. So in Canada, you I don't know, you've got something like 17 banks. In Germany, you've got, I don't know, 30 handful. I can't remember exact numbers, but uh in the United States, you have like 7,000 plus another couple, 3,000 credit unions, right? So we're just the scale, the number of organizations that have to make these changes and adopt this technology definitely does um you know put some friction on the adoption of these technologies. And you know, in Germany and and you know, it's a lot of European countries, Scandinavian countries, uh the government public sector plays a bigger role too in driving uh standardization, driving um uh even you know technology and payments and financial level standardization. We we leave it much more to the private sector in the US, which definitely has advantages, but then yeah, of course, has some some disadvantages as well. So sure, sure. Like I can't have contactless uh I yeah, and yet I soldier on.

SPEAKER_01

Um that's right. Yeah, and I you're very brave. I seem happy enough with the arrangement.

SPEAKER_00

The other the other neat part of uh and the one that's a little less um customer focused in the payments world is uh sort of just bank-to-bank payments. And and we don't talk about this as much. It's not, you know, it's not quite as sexy as credit cards and you know, because it's not in in the uh public consciousness as much because you're not using it every day as a consumer. But um in this country, we use a system called ACH currently, that's kind of the current standard. And again, it's it's just uh slow. It's you know, literally days. Yeah, literally days. And so there's you know, for a lot of consumers, the way that they feel that is on the first of the month and on the 15th of the month, there's this period of four or five days when I just like really don't know how much money I have in my checking account, right? Because my my paycheck's hitting, but then my rent check and my utilities checks are clearing, and it's kind of like, I don't know, I'll wait till the fifth and then I'll I'll figure out how much money I've actually got and then I'll start, you know, buying stuff.

SPEAKER_01

So yeah, you have a like an almost week-long period of eventual consistency. Yeah, exactly. And depending on your financial situation, that can be a harrowing period of eventual consistency.

SPEAKER_00

Oh, absolutely. And and a lot of folks, unfortunately, at the at the lower end of the income spectrum, that becomes a real challenge. And uh the uh a group, I can't remember which research group it was, just did a study and found that a lot of low-income earners end up falling back on bank overdraft fees as essentially a very high interest rate loan to carry them through this kind of five-day flux period. Uh which is which is terrible, right? That's that's not the the right financial uh move to help people to get ahead, right? They're just paying way too much than you need to for that kind of stuff. So it's a bad situation.

SPEAKER_01

So uh it's uh you you mentioned that, so I suspect this will come up in the rest of our conversation. But um you uh started evidence systems a couple years ago, and you are, I believe, bootstrapping and have been doing some professional services. Is that correct?

SPEAKER_00

Yeah, that's right. So um left capital one, um, had had a a couple of um contracts waiting for me, and that's why I felt safe to kind of make the jump. Uh and have done so when I when I did it, that's how I did it.

SPEAKER_01

You that's right, that's right. You don't want to jump and then go look for work for six months.

SPEAKER_00

That's right. You gotta look before you leave. Yeah, so uh had a couple of great contracts, um uh and and I owe you know good friends and um and and fellow colleagues uh a lot for for setting me up with these really great uh uh engagements. So we did professional services for about 18 months, uh including a couple of engagements serving alongside Confluent as a as a consulting partner to Confluent and a couple of uh very large financial uh services firms in the Midwest. Uh and that work's been really interesting.

SPEAKER_01

That's that's been that's been I should say I want to dig into some of that, but I should say so everyone listening knows. Um I actually didn't know that before we started recording today. Um I I uh knew that Bobby had a startup. I knew there was a product happening, we'll get to that. Uh spoiler there, but um I did totally didn't know you were a consulting partner. I think that's awesome. Uh, but just so everybody knows, Bobby is here because he has super cool things to talk about, not because he's a partner and we're trying to promote him being a partner, but by all means, tell us about what you're doing, uh whether it's with Confluent or not. What what are some interesting things that you've done with event streaming in the past two years?

SPEAKER_00

Yeah, yeah. So um probably the most interesting project is the one we I alluded to earlier. Large professional service or professional, excuse me, a large financial services firm in the Midwest. Um they had been uh, you know, a couple of their um senior engineers and architects had been very forward-leaning on event sourcing and um wanted to build kind of the next generation tax management platform. They do a lot of sort of tax stuff, tax accounting and that kind of stuff in their financial services. And they wanted to build their next generation engine on top of this sort of event sourcing architecture with Apache Kafka. And they already had a working system that kind of worked this way. Um, but it was kind of you know half, you know, maintaining current state and and and using the database as the um canonical system of record and then kind of half using events to do that. They had Kafka in the stack and and and they they just wanted to make that system better. The next time they built this system, they wanted to really do it right. So they had seen one of my talks um and and contacted me and asked me to uh just kind of come in and educate them and help them, you know, how do we do this? What types of architecture are we gonna need? Uh in the course of doing that, I discovered that they were also in conversations with Confluent. And so I was upfront with them. So, hey, I'm a Confluent partner. I, you know, I'd love to talk through that and see how we can how we can all work together. So um that ended up you know solidifying and and uh we're engaged there and kind of their whole next generation platform is gonna be um built event first, and you know, publishing the event is is sort of the key factor within the transaction boundary uh for any given command that comes into their system. So uh it's really great. It's a it's a really cool system. They're they're doing really neat things. Um we use some domain-driven design uh techniques, some event storming and event modeling uh workshops with with big groups of uh both technical and business stakeholders there, and sort of really designed out the system that they want to have. And uh and that helps us sort of discover these bounded contexts, these kind of areas of um causal interrelatedness and causal independence, right? So this causal causal areas.

SPEAKER_01

What's that? Causal coherence were uh Yeah, causal coherence, yeah.

SPEAKER_00

So that's how we defined our our bounded context is we said, okay, things that are immediately causally related and that we need to get totally ordered on a single log, that's going to be a bounded context. And you know, we sort of mapped out the the three or four or five different bounded contexts that we knew would need to interrelate in order to build the system. And then we we looked at the commands that came into those bounded contexts, we looked at the events that those contacts emitted, we looked at the read models that those uh contexts would need to build uh for themselves to maintain you know transactional consistency and invariance, and also for other uh consuming bounded contacts to be able to read and see what their current state was, et cetera. So uh it was really neat. It was a it was a great exercise. We had a lot of engineers, a lot of business people involved across a lot of different organizations, and and this has become a really, really nice way to um envision the system and organize around its main parts or main you know constructs uh in order to get the system built.

SPEAKER_01

So that uh uh not technology stack, but uh sort of method or design architecture stack, you know, set of set of tools you just described there uh for building a system. That is, I think, to a large degree, living the dream of event streaming. Like that's the um, I'd say that's a very faithful representation of what everybody sort of affirms as the orthodoxy right now. Like this is if you could if you could do it all the right way, this is what you'd be doing. And you're doing that. Is that have you is that the first project where you feel like that's everything has come together, or do you see that in your practice? Uh is that a big part of your practice, that whole dream?

SPEAKER_00

Yeah, so so that that's the dream we want to live. So anytime we engage, like that, that's how we're gonna encourage our our clients to build their system, you know, not just because it's the you know, the current mode or any of that, but but sincerely because we believe this is the right way to build systems. And you know, I've built a lot of systems. I've been burnt in the past by, you know, you build and build and build and build, and another team over there builds and builds and builds, and then you try to like connect the two sides of the bridge, and and you're off by meters and you have to do all these crazy hacks and cluges. And you know, having been having lived that nightmare sufficient numbers of times, I I really sincerely believe like this is the right way to be able to cooperate without having to coordinate in these sort of large um large uh organizations that have lots of teams that you know you don't want to have to coordinate. That's really what slows everything down. If you can cooperate and observe each other's happenings, you know, at the business event level, you don't have to coordinate in order to get that level of cooperation, which is which is definitely the dream.

SPEAKER_01

So Right, right. That's a great insight. Cooperate rather than coordinate. Could you uh describe that a little bit? Describe what you mean?

SPEAKER_00

Yeah, yeah, yeah. So um and this brings us kind of full circle to that blog post again. Really the the driving um uh organizational design uh level reason for adopting something like microservices is coordination avoidance, right? The point of it is not micro. It's not just to build a bunch of tiny things for the sake of tininess. The idea is we operate in these organizations, we have different reporting structures, we have different bosses, we have different priorities and different needs. And somehow our stuff has to work together. Somehow we have to integrate our views of what's going on in this business. Unlike the public internet, where there's different techniques and different architectures that solve that problem because there's low trust, within this company, we basically trust each other, right? The accounting department basically trusts the marketing department, which basically trusts the product engineering department. That's a terrible idea, but they do. Yeah, right. It doesn't always work out. And some in some environments are higher trust than others, you know, candidly. But um yeah, solving that problem where each team can design, build, and deploy and operate their own capabilities independent of anyone else. Uh, you know, that's the goal. And that's you know, driven by a lot of sort of DevOps values. And so that's why people look to microservice. But most of the time when they look to microservices, they look to this distributed objects. Let's have all these things, you know, synchronously chatting HTTP or whatever to each other all the time. And in that world, you can't reason about the system. It becomes very difficult to migrate and evolve over time without breaking everybody. It's really very brittle. And then reasoning about the state of that system at any point in time is literally impossible. It's sort of non-deterministic. You can't put your finger on like what did we know at this moment? You just can't from a distributed systems perspective. So having a set of methods, like you've been talking about, you know, architectural tools and uh design ideas to allow you to build these independent services and have kind of team scale autonomy while maintaining the ability to reason about the state at any point in time across all these distributed participants, that's really valuable. It's uh extremely um it allows the reasoning about this system, which in uh you know, in other types of architectures and other types of design paradigms is is literally impossible. So, you know, having systems that you can reason about, that's why we called our our company evidence systems. Uh, we want to be able to reason about them. You want to be able to understand what they're doing and understand what they mean. Um most of these ideas are not mine, frankly. These are, you know, living the dream in these respects and doing these consultant engagements. We're definitely standing on the shoulders of giants here. Um great, yeah, one great thinker that I've been reading a lot lately is uh Adam uh Dimitrich. I hope I'm pronouncing that right. Um he's uh he and Greg Young, who's sort of the father of CQRS and event sourcing, are working together on this technique called event modeling. And this event modeling technique I've used to great success, um, and it's really fantastic. It really dramatically simplifies these ideas down to uh you know a workshop with four different colors of sticky notes, right? Four different types of things. And everyone can get involved: engineers, business people, graphic designers, um, architects, you know, everyone can can participate in this workshop. And it's it's just a really great way to um share uh a vision of what you're building and really understand and be able to reason about what you're building. Um but then the artifact you actually build is is a rather rigorous model of a distributed system that you can then implement directly from the diagram. So it's really an interesting technique. I'd encourage anyone who's you know wanting to live this dream uh to look into into Adam's Adam's work. Uh I think it's eventmodeling.com is is the uh is the link there. Eventmodeling.com. No, excuse me. Eventmodeling.org. Yeah.org.org.

SPEAKER_01

I was gonna ask you later. I was just gonna find that, make sure that makes in the notes. Yeah, no, that sounds fantastic. And yeah, certainly echoing your point about standing on the shoulders of the giants. I like every once in a while I'll come up with maybe an original way of explaining an idea or some nice words that go together well uh or something. But the ideas um I mean I I definitely know the sort of graph of intellectual influencers that that have taught me uh how this stuff really works.

SPEAKER_00

Yeah, absolutely. Most of the time when I think I've come up with an original idea, I'm disappointed to find out that, oh no, much smarter people have explained it much better than I. In fact, when I when I was writing that blog post for you all, I you know, I was so proud of like, oh yeah, microservices and Kafka. No one's thought of this. And then I went back and read you know some of Ben Stopford's amazing work on the subject. And I was like, oh, Ben's already covered it more and better than than I could. And you know, anyway, so hats off to him on that work. It's either Ben or it's Martin Kleppman. Martin, yeah, absolutely. Martin, he did the you know, database deconstructed, which actually really kicked off this journey for me as well. Um, Martin's ideas, in addition to like studying Datomic, you know, from Rich and Stu, that was really what crystallized a lot of this in my head when I was at Capital One. So you know, hats off to Martin uh too. His his latest paper on uh online event processing is fantastic. It's really great work.

SPEAKER_01

So yeah, no, it is. I I there are some Martin ideas that at the risk of overstatement, I will not think about software architecture the same way again. Yeah, yeah, absolutely. That's a good thing. That's like you want you want those things to happen to you every few years. Yeah. Um any other uh interesting uh professional services stories, outcomes, uh is this stuff clicking well?

SPEAKER_00

Yeah, no, we we did another really interesting project with a uh major auto manufacturer. Um and and actually, you know, sorry, I know I'm on the I'm on the the streaming audio podcast, but uh we actually ended up using Redis and Redis streams instead of Kafka for that particular um uh project because they already had Redis in the stack and they were comfortable operating it. And so we're like, okay, fine. And you know, uh Salvatore had just added streams to uh to Redis. And so we ended up using Redis and Redis streams, which was uh cool. I mean it was it was different for me. I'm I'm you know, 80, 90 percent of the time I'm working in in the confluent stack, but uh it was interesting and and different. And I was able to speak at RedisConf and Redis Days New York and a few other places about uh about that system. So that was cool too uh from a professional services standpoint.

SPEAKER_01

And that is not at all a problem to mention on streaming audio podcast. By the way, there are technologies that are not uh confluent things, that are not Apache Kafka, and they're very interesting. Uh recently recorded an interview about Apache Druid uh that probably has aired. Uh never quite sure about uh uh the ordering of these things. It's probably aired uh already at the time that that uh you're listening to this. But yeah, I mean it's uh it's a big world. And one of the things about Kafka is it um well, there are various ways of looking at it. One of them is as glue, and uh if it is glue, then there are lots of other systems that you're gonna care about, right?

SPEAKER_00

Yeah, yep, absolutely. Um we've spent oh sorry, uh Tim. Yeah, we spent a lot of time doing professional services. Um now we're sort of in this uh a little bit of an awkward adolescence where we're we're still doing professional services work and it's still very important to us and we want to you know keep those clients happy. But we've also just launched our first product. So we're in that like awkward adolescence of a startup where we're we're trying to make that pivot.

SPEAKER_01

Yes. Tell it tell me about that. That's wonderful, by the way, that that you're doing that. But uh the product.

SPEAKER_00

Yeah, so uh we launched our product at uh Finabate Fall in uh New York City. Um, I guess, oh, I guess it's been about a month ago. Wow. That month went by really fast.

SPEAKER_01

Um by the way, we're recording this towards the end uh last few days of October 2019. Uh so a month ago as of then.

SPEAKER_00

Yeah, yeah. So it was the I think the 23rd to the 25th of September. Uh and and Finivate's a really great show. It's uh it's all demos. So if you if you get on the the big stage at Finovate, you've got seven minutes, you can't show any slides, you can't show any video, you've got to show working products on the stage. Uh it's fantastic. So you can end up with yeah, right? You can end up with really fantastic demos. And you know, the vast majority are are super polished and really great demos, but I mean it's just the potential for like spectacular people, you know, falling on their faces and the demo not working. And you know, it's it's a it's an interesting show.

SPEAKER_01

So I'm as a guy who live codes, you know, sometimes it happens.

SPEAKER_00

Yeah, you know better than most the hazards of live coding. I've seen you do that at uh at various shows. Um so it's a great show. We had a great time. Um I think our our demo went really well. The video is available online, uh, which I guess we'll probably link to in the show notes. But the the product is um I was talking about payments technology earlier, ACH and kind of how it takes two or three days, and it's a pain. Um very recently, the Clearinghouse, the same company that sort of maintains a lot of the ACH infrastructure in the US, uh, the Clearinghouse, it's a consortium of, you know, it's a joint venture, a nonprofit joint venture of some of the biggest banks in the country. That's what it's actually called the Clearinghouse. The Clearinghouse, yeah. It's it's a very old company. It's been around since like Yeah, right. It's been around since the late 1800s, I think. It's very old. And uh and it's all about just moving money among these banks. And how can we move the money around among these banks, right? Um and so they've they've maintained various infrastructure over time. Um ACH is about 40 years old and it's starting to show its age. And so they uh about I guess five years ago started working on a real-time payments platform. And real-time payments is just that it's enabling money m money movement among US banks and credit unions in real time. And again, we're sort of embarrassingly behind Europe and Asia in in these ways, and even South America. A lot of uh other countries already have real-time payment infrastructure that allows you to move money between banks in real time, but we don't. Um, so this is really groundbreaking. Um, additionally, you know, in addition to just kind of the money movement capability, um, there's also an information movement. So you can have information about the transactions flowing, you know, right alongside the transactions in the protocol. Um, so it allows for really rich descriptions of um business processes. So you can move money and then you can send a message saying, hey, remember that money that I sent you? That was for you know, purchase order number or whatever, and invoice number, whatever. So you get these uh more business level view into these transactions right within the payments protocol. It's super powerful. It's a really, really well-designed system. And so our first product is um enabling banks and credit unions, in particular smaller banks and credit unions, to get on to uh the clearinghouse's real-time payments network. So we've partnered with the Clearinghouse. Um, they were on stage with us actually at Finovate, um, you know, kind of introducing us. Wow and uh so we partnered with them and we're helping banks sort of get involved in this payments network. Okay. So it's it's really great. We think it's a great opportunity for smaller banks and credit unions to uh compete and stay, you know, technologically even with the bigger banks, uh, as well as provide just a really great service to their customers.

SPEAKER_01

Uh yeah. No, so I I um by the way, I didn't know there was an organization called the Clearing House. And I'm glad I know that now. And um I didn't know there was a real-time payment system under development. And I'm guessing if you're a big bank, you know, there's uh spec uh and protocols and you know, all kinds of things that you you have to implement and you have you know people who understand the spec and it's their job to just sit there and read that thing. And if you're a little regional credit union with an IT staff of five, you can't do that.

SPEAKER_00

That that's exactly the problem. And you know, the the clearinghouse is a joint venture of the 25 biggest banks, and so as they were designing this protocol, it was sort of obvious to them how to build it for themselves and for each other. But you know, I don't know if they were thinking about the the smaller banks as much in response.

SPEAKER_01

I don't know anything about financial services. I'm just gonna venture a guess. Maybe they actually were thinking about not thinking about them.

SPEAKER_00

So yeah, that's possible. Um for the most part, the cleaning house does a really, really great job of of staying neutral and staying very accessible. In fact, like the uh the fee structure and and and you know what it costs to participate in the real-time payments protocol is like super cheap. I mean, like orders of magnitude cheaper than any of the other you know, wires or ACH or any of these other things. So uh they they were designing it to to be very accessible, but uh yeah, I mean the just the technical hurdles uh are are challenging, right? You have to you have to have a hardware you know encryption matched router in your data center, and it has to be actually in two different data centers, and you have to, you know, there's a lot of sort of um you have to be this tall in order to participate uh types of technical uh hurdles. Absolutely and and yeah, just just the software engineering resources to implement the spec, like you're saying, you know, that alone would be enough, not to mention kind of the operational and hardware side of the things. So um, yeah, it's out of reach for a lot of these smaller banks without a third-party service provider. So we are a third-party service provider within the network, and that's like a defined entity within the within the network. Uh, and so we help these smaller banks and credit unions. We've written the connector, the connectors into their core systems of record, so they don't have to write any code at all. We just sort of they log into our tool, they fill out a couple of forms, and then they're you know up and go ready to roll from a test perspective, and we can test and certify their integration, and then they can go live uh with with the uh integration all right from our dashboard. So it's it's much more a product experience for them, like kind of like a SaaS portal experience, and they don't have to worry about the software engineering side of that. We've taken care of you know the operation side and the software engineering side for them.

SPEAKER_01

So absolutely. And you've uh so the the hard problems there are those things, right? This this special encrypted hardware and all that. What what else just from a top level, because you've been doing payments for a while now. Um, if you could just tell somebody payments are hard because, you know, what is what is that? Why are why are payments hard?

SPEAKER_00

Yeah, yeah. Um so so payments are hard generally, and and this particular jump from a batch-oriented payment system in ACH to a real-time payment system is is even harder. And I'll talk about that a little bit too. Um, so payments are hard because for all the reasons we've talked about, you have to implement a common spec, right? By definition, you know, payments and the protocols that enable payments are very much like the World Wide Web and the protocols that enable the World Wide Web, right? You have to know HTTP and TCP IP and TLS and you know, all these things to participate in the World Wide Web. Um you go down to a uh something as sensitive as payments, and and there's a lot of security stuff, there's a lot of um redundancy and failover stuff, there's a lot of um you know weird, tricky edge cases. You know, what if what if the person that you're trying to pay has died and their account is in receivership with someone else? What if you know right? There's all these super weird little edge cases like that that you have to deal with and you have to be able to respond in a in a sane and legal way. Uh that's the other big part of is uh the the laws and compliance's compliance um and governance sort of burden around payments is much higher than it is, say, you know, in the web case where you're talking about showing cat pictures on the internet. It like when you're talking about payments and people's money and people's life savings moving from their investment account into their savings account so they can start drawing it down during retirement, that's a huge deal, right? If you lose grandma's money, someone's going to jail or someone's gonna get fined or something, something bad's gonna happen. So uh just operating within the compliance and legal framework is very tricky as well. So that's kind of all the reasons, or not all of them. That that's uh a surface treatment of some of the reasons why payments are difficult. Moving from a batch payment system like ACH into a real-time payment system is even harder because it it changes, it kind of breaks the frame that banks are used to working in. So banks for a long time, I mean, we're talking like since the 1500s, have operated in this um kind of bankers' hours mode, right? When the banks open, you can transact with them. When they're closed, you can't. And that's for a good reason. When they're closed, behind the scenes, they're doing important things to make sure that they can open the next morning and that they'll be solvent and that they have enough cash on hand to satisfy their likely um uh you know depositors coming with to withdraw funds and all those things. So um this idea of bankers' hours where we're open for some of the time and then we're closed and doing important things behind the scenes for some of the time, this is a very old idea. It's been around for centuries, right? Right. And and now moving into this world where you're talking about 24-7, 365 on, constantly being able to move money around in real time, just providing kind of the you know, the the liquidity fuel to make that whole thing go, is just it's a new idea. It's a new idea for banks and it's challenging. Um the idea that a customer at two in the morning can, you know, be trying to send a payment and then get mad because they can't send a payment and want to call someone and want help with that. Uh this is just not the way that most banks operate. And so in addition to moving from a real-time or you know, from a batch mode into a real-time mode from a payments perspective, they've also got to do that from a customer service perspective and from a monitoring and operations perspective. And right, there's a whole bunch of things that kind of, you know, payments is the camel's nose under the tent door, and then you know, the rest of the real-time camel is gonna be in there pretty soon with you. So um we want to help, you know, payments is our first product in the evident real-time suite. It's called Evident Real-Time Payments. Uh, but there will be other products in this evident real-time suite that will help banks make that sort of holistic transition into real-time from a monitoring and customer service and uh, you know, other things perspective. That this is a bigger change than just payments. And I think banks are going to discover that as soon as they uh adopt this and operationalize it, they'll realize, oh, you know what, there's a lot of other things we can do from a real-time perspective. And you know, maybe there'll come a time in this country when your bank's on 24-7 and there's no bank holidays from a from a service perspective and from a uh you know customer uh availability of my money perspective, you know, maybe we'll get to the point where money just moves and you see the current state at all times, and there's not these sort of blackout periods where, oh, Monday morning a bunch of transactions hit, and now my financial situation that I can observe is significantly different from how it was Friday afternoon, right?

SPEAKER_01

Yeah, yeah. Man, and I mean you're actually working to make that happen, which is fantastic. You remind me though, of so you know, there's a transition happening from batch payments to real-time payments. Uh, okay, fine, that's technologically difficult. Um, you need some new skill sets. There are people like you who can develop products in the ecosystem to help uh smaller players get on board, the larger players are building out practices and you know I see that from the the technical people at large financial institutions who are in fact doing uh state-of-the-art advancing stream processing work, right? So like those those large uh financial services institutions have built that expertise and are frankly pushing a lot of the community along having built that expertise. So that technology transition is happening. Yep. But I'm reminded of a different topic, but a a talk uh Kent Beck gave. And this was at a conference in, I think, Gothenburg, Sweden, uh a number of years ago, but it was um a little bit early in the discussion about continuous deployment, um, when that was a thing that hardly anyone was doing and this revolutionary idea. And his talk was all about you know, imagining you have a slider from annual releases to actual continuous deployment, you know, merge to master causes tests to run and build to deploy and all that. Um he took like four or five steps along that slider and said, here are the cultural and organizational changes that are implied by a position there. Uh and it's totally true. And like, you know, now we can tell a thousand stories about organizations going through that transition. But it strikes me that this is gonna happen. I mean, you just kind of described that possibility. Uh, once payments are real time, um, that's not just a difficult technology change. Like, I kind of trust that everybody has tooled up and hired the talent and built the talent and we can build this. Uh, but that now, you know, now that bankers' hours become not a thing, there are just a lot of other cultural changes that seem like they'll attend to that. Not that we want people to work 24-7 and and you know, this become some terrible staffing thing, but just the way you think about your business and the way you operate is gonna be very different.

SPEAKER_00

Yeah, absolutely. And yeah, that's right. That's right. Um, and a lot of other industries are going through similar transitions, right? I mean, that this isn't this isn't unique, but there definitely is a a sympathy and an interplay between technology and organizational design and and and culture, right? And and they influence each other. Um and and the the design of your technical system will come to resemble your organization chart. And the design of your organization chart uh will be influenced by the you know the the technical systems you have, right? So there definitely is you know this this interplay among those uh among those things. And this I think is an example, just like CICD, you know, and some of the DevOps values are driving, you know, one type of cultural and organizational change in IT departments. I think this uh just exactly to your point is this real-time idea is gonna move banks along. And I think it's facilitated by technologies like Apache Kafka and uh these other sort of real-time enabling technologies, right? Before we had these technologies, you it wasn't even worth talking about trying to do real-time, you know, 365, 24-7 banking or anything like that, because you know, frankly, the infrastructure and the cost of trying to support that is outrageous relative to the to the value to your customer. But with these technology changes, that the economics change. And you can start talking about a real time banking infrastructure. You can start talking about, you know, 24 7, 365 access to my money and and all that. So yeah, these. Enabling technologies are going to definitely drive organizational changes in these banks and operational changes and economic changes. And that I think it'll feed back eventually the other way around, where you know these changes, once fully adopted and really kind of institutionalized in the in the DNA of these financial institutions, they're going to reach back out and start looking for more enabling technologies that are going to help you know continue that virtuous uh kind of feedback loop. Um so we're sort of we're seeing the early days of that now. Um some of the bigger banks are are doing it from a technology perspective. Um but yeah, in the US especially, you've got this really long tail of smaller banks and credit unions that that are essential and and critical to our economy. They're part of why uh the US banking system has been as robust and as um uh economically valuable as it has over the past couple of centuries. Um so the fact that we have this diversity uh uh of banks and you know that provides some level of kind of fault tolerance, right? If you think about a distributed system, um we need to maintain that. And and so enabling small banks is important to me, you know, from a from sort of uh globally what's what's best perspective. I think community banks and regional banks, credit unions, I think they they play a really important role. And so I want to help them, you know, kind of maintain parity with the biggest banks, to be able to provide the unique personalized customer service that they do for their customers, to be able to do small business lending, which they do most of in this country, uh, to to their uh constituents kind of in their same geographies. Um all those things are important. And so we need to be able to sort of enable these smaller banks and credit unions in that way.

SPEAKER_01

So uh that would be an amazing closing note, but there's and I I really appreciate just kind of your philosophy of approaching this. I think it's fantastic. Um there's one other thing I want to ask you. Uh, and there could be people uh who who are new to the podcast, they're like, oh, Bobby is gonna be on there, he's gonna talk about closure, and they've been waiting like 45 minutes now for you talk about closure. And I it would be, I think, rude, not just to that new segment of the audience, but to you. Not to ask you to tell us tell us about the closure things you're doing. I mean, I know that's what the stack is built on, but what uh just give us a walkthrough for those who are fans.

SPEAKER_00

Oh, great. Yeah, thank you. Uh thanks for bringing me back to that. You know, starting to wax philosophical and like being real high level here. Uh yeah, let's get down to like the cool nerd stuff. That's what I like to say. Tell me about the library. That's right. That's right. Um, so closure and functional programming generally is built on this idea of having um few data structures and many operations on those data structures. Stu Holloway's talked at great length and much better than I could on the subject. You know, object-oriented programming has kind of the inverse uh orientation where you have a ton of data structures, right? Every class you define is kind of this unique data structure, and each data structure has relatively very few operations on it. So functional programming has this inherent sort of composability where you know, because functions are first class, I have higher order functions that take functions as arguments and do interesting things with them. Um you can sort of build up your whole the business logic, the functional core of your system. You can build up just by tinkering with functions and just playing with functions. And in most systems that I build and you know, a lot of functional programmers sort of advocate this idea of a functional core and an imperative shell. So you've got these concerns at the edges, these side effects where you're you know putting data on a socket or you're writing something out to disk or you're talking to the user or getting input from the user or sending an email or you know, these kind of side effects, concerns at the edges, but then in the middle you've just got these pure functions and and they encapsulate your business logic and they're easy to test and they never change. And you know, you have this um, you know, again, this determinism where every time I pass the same arguments to this function, I always get the same answer back as a return value, and there's no side effects caused. And you know, it's just easy to reason about a system like that. So Kafka and Kafka streams really lends itself well to that. Um there's a notion that was sort of popularized and uh made very accessible by Rich Hickey called a transducer. And a transducer is um, without getting you know too far into the technical weeds, a transducer is a way of encapsulating the processing of some sequence of values and incorporating them into uh a collection, a composite value. So you have the operation reduce, um, which is you know, I've got uh an initial collection, I've got some collection of things, and then I have a function. And the function job is to take a single element from that collection and incorporate it into this composite that I'm building. So the idea of a sum is a very simple form of reduce, right? The function is just plus, and the sequence is a sequence of digits, and I reduce over that collection of digits, and I and I add them together, and I incorporate each one into the composite by adding its value to the to the sum up to that point that I've calculated. So that's a simple reduction. Uh, but you can do that you know with all kinds of different things. Um, you know, reduce is uh is a very primitive, very powerful thing in functional programming. But if you take away the source of where I'm getting each individual value and the destination where I'm putting it, the composite thing that I'm building, you're left with just the reducing function, the thing that you know does the job of taking each element and incorporating it into that composite. If you can separate those things a little bit, then you end up with a transducer where you say, okay, I don't care how values arrive to me, if they arrive to me on trucks or if they arrive to me on conveyor belt or if they arrive to me in airplanes, I'm still gonna do the same business logic and I'm gonna put them on the destination truck. But again, that destination truck can be something else too, a train or whatever. So that's sort of what a transducer is. There's a lot of really great writing. Don't you know take my hodgepodge uh description of transducers here as gospel, go read some of the source literature and uh on the closure.org website and so forth. But having a transducer is really powerful in a in a uh Kafka streams sense. So uh if I've got this business logic that allows me to take a value as it arrives, you know, say from a collection in memory or from a Kafka topic, I can do something important with that value. I can I can you know build up a composition of functional transforms on the on these values, and then I can put it somewhere. Maybe it's in a destination collection in memory, or maybe again it's on an output topic from Kafka streams. What's really valuable in Clojure is that you don't have to re-implement your um your functional transform language for each context you have. So whereas, you know, the really excellent, you know, and I don't mean to demean this in any way, I think Kafka Streams is a really awesome piece of work, but the high-level API has operations like map and filter and reduce and some of these things that, hey, I've already got that enclosure. I've already got that, you know, as native functions in my language. Why do I need to use the API just because the context has changed? The fundamental logic is the same. So what uh a couple of libraries that that we're using now, one's called Willa. Um, I wrote one ages ago for using uh transducers and and adding Kafka streams as just another transducible context for closure transducers. It's just a thing that brings values to my front door. I still use my regular language of function and function composition to do the business logic piece, and then I have a new context for where to put those values. You know, it's an output stream or whatever. So uh using closure transducers in a Kafka streams context is really powerful because it allows me to just use my native language construct. It's just functions, and I can test them the same way that I test any other functions. And then when it's time to run the system, I actually just hook it up to a new context, a new transducible context, which is Kafka streams. Values arrive on topics, I perform my business logic, which I've tested in other ways, and then I write those values out to destination topics. So that's been really powerful using transducers in our Kafka streams um topologies. It's really easy to reason about, and we don't have to use a lot of the sort of um testing infrastructure that that other languages have to use um for testing, you know, like the topology test driver and some of those things that you have to use from Java because Java doesn't have any other better options. In Clojure, I just test my functions the same way that I test any other function.

SPEAKER_01

And it just semantics of you know, say the reduce function are well, they're the semantics of the reduce function. You already know them. That's you're not learning a different API that does a conceptually similar thing in a different way.

SPEAKER_00

That's right. So I can just build a composition, you know, I can say for all you closureists out there, I can say comp, you know, map whatever function, filter whatever function, uh take whatever function, leave whatever, you know, I all the different sort of um transducers that I have in the in the standard library. I can build a functional composition representing my my business logic, my sort of computational pipeline. And I don't have to do that via the Kafka Streams API, I can just do that with normal functions. Yeah, and then I just plug the result. The result is basically just my transducer, and then I plug that into uh into Kafka Streams via Willa, and it just sort of works. So we're using Willa. Um Jackdaw is another great library that that is sort of a fellow traveler, I think, with with the Willa library. Um but Jackdaw's a really great piece of work uh for using Kafka and uh some of those other things from a closure context. Uh and then we're using uh Kafka Connect. We use the Kinect API to uh um bridge you know the the payments protocol into Kafka topics and then from our output Kafka topics back into the protocol. Uh we're using Kinect on both of those edges, and that's been really fantastic to work with. Um we're storing state just in our streams jobs. At this point, we don't even have a database. We're just using uh uh you know the the event stores um and global K-tables, you know, built right into Kafka streams, which has been really nice. I think eventually we'll end up with a database just because you know there's some uh operational advantages there. But uh up to this point we haven't needed one, which is really neat. So we're really far along in the tool development and uh uh implementation of this payments protocol, and we haven't needed to reach for uh for a database yet. For a database yet.

SPEAKER_01

I agree, by the way, I we are far from so doctrinaire on that point that we would suggest that you never need a database. I mean you're gonna need a database at some point, but um, I think doing you know, as we said before, living the dream as you are, uh you do get farther down the road, and there are lots of cases in the margins where database is all you had, uh, and so you used it, but now it's not where you go to put state. Um there still are things that you're you're gonna need it for. Um but uh you know, in the margins, you've got these other tools that that fit better, fit the rest of your paradigm better.

SPEAKER_00

Yeah, exactly. And and there's there's some subtlety and some importance in that uh there's there's more philosophy that you can unpack in that decision too, because it used to be that the database and its design and its affordances really influence the design of your system. And now we're talking about designing and implementing whole systems without even thinking about or reaching for a database, so you know that it's its affordances and its idea of the world isn't influencing the design, right? The design is what design needs to be. And then the database is sort of a late-bound decision that you can throw in there at the last minute if you need it, you know, as an operational uh nicety or for querying, you know, speed or or whatever, but but it's not influencing the uh the design of your system like maybe the systems of 10 years ago where the database is the center of the world and it its opinions really irradiate out and propagate out anything in a bad way up into the design of your system.

SPEAKER_01

My guest today has been Bobby Calderwood. Bobby, thanks for being a part of Streaming Audio. It's been a pleasure, Tim. Thank you very much. And there you have it. Before I go, I want to tell you that we have a pretty cool new offer to help you get started with Confluent Cloud without you having to pay for anything. If you're a new user and you go through the regular sign-up process and start using Confluent Cloud, your first $50 of usage per month are free. This will last for the first three months after you sign up. So that's $50 per month of serverless Kafka for three months at no cost to you. So go to the sign up link in the show notes. I don't want to read you the URL, and sign up now. I think the only thing I could really do more is write your code for you. And I think we can both agree that's too much to ask. So check it out and hey, let us know how you like it. Anyway, as always, I hope this podcast was helpful to you. If you want to discuss it or ask a question, you can reach out to us on Twitter at Confluent Inc. or reach out to me at TL Burgland. That's T-L-B-E-R-G-L-U-N-D. Or you can hit us up in Community Slack. There's a sign-up link for that in the show notes as well. And while you're at it, please subscribe to our YouTube channel and to this podcast wherever fine podcasts are sold. And if you subscribe through iTunes, be sure to leave us a review there. That helps other people discover the podcast, which is a good thing. Thanks a lot for your support, and we'll see you next time.

Head Shot

Tim Berglund

Host
Head Shot

Adi Polak

Co-host
Head Shot

Viktor Gamov

Co-host