In this episode, we will be discussing the business of software development. Writing software doesn’t happen in a vacuum, it often happens in the context of a business, and it can be useful to understand exactly where software fits in a business.
To explore this topic, I sat down with Q McCallum, a data consultant who started his career in the software world (and who, somehow, managed to never really leave it). His consulting practice focuses on on guiding companies on their first steps in data science, machine learning, and AI. With more than two decades of experience dealing with a variety of software companies, Q has accumulated a wealth of knowledge on how these companies operate. While our conversation started on the business of software development, we ended up gravitating more towards discussing the realities of software development (which often happen in a business context).
In this episode, we will be discussing the business of software development. Writing software doesn’t happen in a vacuum, it often happens in the context of a business, and it can be useful to understand exactly where software fits in a business.
To explore this topic, I sat down with Q McCallum, a data consultant who started his career in the software world (and who, somehow, managed to never really leave it). His consulting practice focuses on on guiding companies on their first steps in data science, machine learning, and AI. With more than two decades of experience dealing with a variety of software companies, Q has accumulated a wealth of knowledge on how these companies operate. While our conversation started on the business of software development, we ended up gravitating more towards discussing the realities of software development (which often happen in a business context).
Hey everyone. Welcome to conversations in software development, a podcast on software development intended primarily for students. I am your host. Paul has Sotomayor. Uh, I am a senior lecturer in computer science at the university of Chicago, where among many other things, I teach a class on software development. In this episode, we will be discussing the business of software development. Writing software doesn't happen in a vacuum and often happens in the context of a business. And it can be useful to understand exactly where software fits in a business, uh, to explore this topic, I spoke with que McCallum, a Chicago based software consultant whose current focus is on guiding companies on their first steps in data science, machine learning and AI. Uh, but with more than two decades of experience, dealing with a variety of software companies, uh, Q has accumulated a wealth of knowledge on how these companies operate. Uh, so I wanted to pick his brain on this subject now, uh, Q is one of my favorite people to chat with about software and we ended up talking for more than an hour, which I managed to edit down to about 25 minutes. And while we started on this subject of the business of software development, I'd say that our conversation ended up revolving more around the realities of software development, which often happen in a business context. In any case here is my conversation with Q McCallum. So how about you tell us a little bit about yourself Q
Speaker 2:Uh, you know, I would, I would love to do that. I think, uh, when you, when you explained this podcast to me, you said that it will be released to the general public, but it's also very much for your class. I'm sorry to bring that up because there are of things I could talk about in relation to who I am in the professional sector, but I think it's best to focus on the things that would be most relevant and interesting to your students. Um, I guess before I begin, I'd also like to say, thank you for inviting me to join you in this podcast. You know, you've invited me to guest lecture with your students, I think two or three times already. And they've all been great fun. Um, so I guess where I'm going with this is I am just flattered that you think it's worthwhile to actually record what I have to say. And
Speaker 1:I mean, what do you have to say is interesting? And I think that, uh, every year that you've come to speak to my class, like there's always been students who there's always been at least one student who would tell me like, Oh, my favorite guest speaker was cute. He was really interesting.
Speaker 2:Wow. I think that means the check cleared, which is, which is good.
Speaker 1:Yeah. Money,
Speaker 2:But I mean, jokes aside, I do want to focus on what's important to the students because, um, you know, something I've mentioned before when I've done this in front of the class, just that I I've done guest lectures before for your class and other people's classes and I can empathize, I do remember what it was like being a student and someone comes in and your view is okay. I don't know who this person is. I don't think they have any impact on my grade. I don't see how we're going to really help me get through this quarter. So why am I listening to you? Because you've got five minutes, right? Like it's, um, it's even scary that I stand up comedy audience. So it gives you, at least there's a comedian. They kind of know who you are and you get the applause and they'll give you a few seconds, really start yourself in motion. And after that, it's just, all right. It's the Wolf's. Um, so as far as what's relevant to your students, I mean, this is a course on software development and I have been, I've been doing some flavor of writing software in a professional sense, uh, for more than two decades. Now I started my career back in the late nineties, but I will also say that even before that, um, as a kid, I think since like the single digit ages, I was writing software in my parents, computer learning, I think way back and it was basic and something else. So it really should have been obvious to me before I got my university degree that I would end up doing something with technology and writing lots of software, because that's just what I've been doing for so long in my life. Um, as far as what I've done professionally, uh, since the.com era started off writing lots of Pearl for web development, that's cause paralysis be hot language back then for any sort of text processing or web development. Also writing a lot of tools for systems management, because I was working in places where we had to write a lot of our own tools just to take care of day to day things with the machines later got into what applications and asynchronous messaging, lots of enterprise Java, also writing some trading engines and C plus plus because at the time, uh, and I think even still to a certain extent today, C plus plus has a reputation on, uh, in the world of trading. And today I, my title is no longer officially a software developer I'd say about probably 10, 12 years ago, right? As the term big data was really starting to become a thing. That's when my career shifted. And I was moving away from pure systems administration and pure software development and more into analyzing data, uh, for fun and profit. But what I found most interesting about that and why it's still relevant today, as far as writing software is because a lot of people look at that field, whether they call it machine learning or data science or AI, and they, they assume that it's all this magic that we call algorithms, right. And artificial intelligence, and really a, a good portion of that job still involves writing a lot of code. Right? So that's why, even though I no longer have the title of software developer, I'm still writing on, of code these days. And so I'm very thankful that I develop this skill early on when I did, because it's proven useful throughout pretty much every phase of my career.
Speaker 1:So the, the, the, the subject of this episode is the, is the business of, uh, of software development. And I think that a lot of students don't understand like how diverse, you know, uh, software development businesses sort of can, uh, can be. I think that a lot of students learn students look at software from the outside, looking in and, you know, they're familiar with what they use on, you know, uh, various websites they use on mobile apps, uh, software, they might be running on their, um, and while it is true that there is a lot of software that gets developed, um, that, that gets, you know, you, you work as a software developer and you're writing something that is going to be used by, by end users that is going to be released publicly in some way. Uh, but there's a lot of software that gets written, uh, out in the, in the world that doesn't really fit that, that, that, that mold. Um, so, you know, you having, uh, you know, either work for a lot of companies or in, as a freelancer or consultants, like having interacted with a lot of companies, what, what other sort of like, uh, in what other manners do you see software being, uh, like built, uh, in companies?
Speaker 2:Yeah, I would say to, to answer that let's first start off with, with what you were talking about before a lot of the software students have seen, um, that they've used on their own, for example, everything from an app on their phone, um, to software on their computers that they use to write papers or write spreadsheets, whatever. So let's, let's call that off the shelf software, right? So named, because back in the day, when they were actually physical stores, you can enter to buy software, which sounds, I just stayed at myself. So, and so the, the, the neighbors sort of grew up from that to just refer to any sort of software that was made by some company whose entire role is to make software, to sell to you, right. Like a spreadsheet or word processor, that sort of thing. So that said, um, like, like you said, a lot of companies wanted to create their own software, and there are a number of, I guess, reasons for that. But the overarching reason, uh, the umbrella reason went down to one of two things, right? It's either one, there are some software that we want to use, but it's so expensive that we can't afford to buy it off the shelf. So we're just going to build our own are gonna run equivalent thereof. Right. Um, and sometimes price is a good reason to that, to do that. Sometimes it's a bad reason. Sometimes it's a good reason because maybe there is a piece of software and you only need a small fraction of its, uh, of its capabilities, but you can only buy the entire thing at once. And the cost is just far too much money. I'll, we'll build our own just to fulfill that one particular use case. And we're good. Um, and the other reason to build one software is because there is something that, uh, you want to do something that you want to automate. And there really isn't commercial software off the shelf software that does it. So you have to build it yourself, and this can be everything. Or, and by the way, the sort of sub reason from that one is there is something that you need to do in your company that is just so special. You just found something, very, something that gives you sort of a secret edge over the competition that you're going to have to build it yourself and keep it secret and stuff. Right. And so when, when you think about those definitions of those reasons, why companies will build custom software that opens up a lot of doors, right? That's everything from, um, from trading firms, I've worked in firms where we built our own trading engine that will, um, read and market data and place trades, according to some strategies that our researchers have developed. That's one reason divulge run. One case readability run. Another case is you work in a large company, you have people doing all sorts of manual labor, but more of the manual labor of just lots of shuffling paper around lots of, um, you know, manual entry of, of some sort of data or something. You know, that's the case where you can take a task. That is my, what I call the three pillars, right? It is dull, repetitive, predictable for me. It's all of those. You can probably write some sort of software to automate that work. And what that means is that you free up the humans for far more interesting work or for work. And it's far more, um, I guess you could say work that really requires the human brain more. Right. Um, but pointing back, those are, so those are the big reasons why companies would build a custom software. It's just that they have something they need to automate, and either they can't get it off the shelf, or they don't want to get it off the shelf. That's fine.
Speaker 1:So, so a lot of software developers, uh, you know, end up working on software that, uh, uh, that, you know, in one sense, like never sees the day of light in the sense that, that, you know, it's, it's never going to be used outside the confines of that company, but, but that software is still brings a lot of value to the company, right?
Speaker 2:Oh, it absolutely does. And I'm glad you used the term value, um, because a lot of people, when they think about software, to your point, do you think about a commercial software house and their goal is to make a profit they're focused purely on making money by selling it. Um, when you think about the value custom software can provide, it goes back to what I was saying before about automating something internal. Um, I guess what that means is if you're a company, every company has some sort of business model in it. And that usually boils down to making money by selling something. So one way to make one way for that company to make money is to actually sell whatever is the thing that they make. But another way to make money in an indirect sense is to save money, right? Um, and this is a lot of, this is just econ one Oh one. If you have 50 people doing some sort of very dull paper pushing, they're wanting to make some number of mistakes because as humans we're going to get forward and they're going to drift off and mistakes will cost you something, um, or let's say whatever sort of paper pushing these people are doing. Um, you just see more and more wood coming in. Um, people do not scale well, right? You can say that for I'll just make up numbers. Now, if a human being can handle 50 reports a day and suddenly we've brought on more customers, and now we're going to be seeing 500 more report today, I need to handle, I need to hire 10 more people, right? Whereas software scales, number very different way. You build a software once, and there's an asterisk on the end to come back to that. But you build the software once. And in theory, as long as it's sufficiently efficient and you have, you know, robust and margin of hardware to run it, you can scale up to some near infinite of customers without having to bring in what people that pay, save money, instead of having to hire 10 or a hundred more people, you pass off to the machine. Now, back to the asterisk, before we go on, um, like I said, you hand it off to the machines and you build it once and you're done. Not really. Um, one of the things about software is that it is never really done. There is an old, there was no number rolling around that says about 80% of the total cost of building custom software is in the maintenance, right. Um, a lot of companies make mistake
Speaker 1:And, and I'm glad you bring that up because I have used that exact same sort of like phrase in my class where I sort of tell students software is never done. I think that a lot of people, this is something that you just don't get to experience as a student, because there's always a finality to like, you know, working, doing a homework assignment or doing a project, like when the class is over the classes over, like, you don't have to revisit that ever again. And I think most students don't realize how much time and effort, uh, you know, is needed to maintain software.
Speaker 2:Oh, no. Abs absolutely. And it's funny because one of the things you mentioned just now, right? That as a student, you take all these courses and it usually boils down to, um, you know, make, make a widget or create some piece of code that produces some number X and turn it in by 11:59 PM. And then you never see it again. Right. And I know before we started recording, we were just talking about my early career in working with people from all different walks of life. And one of the things I noticed is that did I meet a number of people who had formal CS degrees who are great developers? Absolutely. I did that. I also meet a number of people who had CS degrees, who didn't fit well and industry. Absolutely. And usually when they didn't fit in it's for precise, where we actually just said they had spent four years learning to write it, hand it in and walk away. And this notion of building software that needs to run more than once building software, um, that you're going to have to extend six ways till Sunday, which means having to write it in such a way that it is extensible. And then it's not very painful to add new features. That's, that's not, that's not the student's fault. That's just something that breaks schools seems to teach at the time when I started, maybe it's changed since then. I do know, um, one of my friends, who's a lecturer at university of Chicago. He's created this amazing course where he gives his students a view of the real world of software development. I'll introduce you to some day. I think you'll like it.
Speaker 1:Who, who, who is that? Let me, let me just write it down so that, cause I I'd be interested in having a chat with that, that fascinating person.
Speaker 2:But, um, but that's one of the reasons I enjoy coming to your class to guest lecture is because you, through creating this class, you're, you're providing a service to, to the professional software industry and that your students leave this class realizing, Oh, it's more than just, I turn it in by 11:59 PM. And I'm done right as we teach them in this course, we'll know we're going to have code reviews. We're going to add features are going to do this, that and the other. And it just gives them a nice taste of the real world.
Speaker 1:Well, and you and you, one thing we did this year
Speaker 2:For the first time was, uh, last year the students worked on a project. Uh, and this year, instead of just deciding, Oh, let's just come up with an entirely different project. We just said, no, we're just going to keep developing the project from last year. So this year students inherited all the code from, from, from last year, which by the way, when I told last year students, like, by the way, I hope you realize that next year students are gonna be working on your code. They were mortified. Like that never happens in a class. Like you never, you, you sort of feel like you're writing code and you feel, well, the only victims of this code are going to be the poor TA. Who's going to have to grade this, this code. And now it sort of dawned on them like, wait a second, like other other students are going to be using this code. And I feel like it's sort of impressed upon them. Holy crap. Like I need to make sure this is actually well-written. It has to be well-documented et cetera. Absolutely. There is a line and I am sadly, I didn't prepare this or I don't know who said it, make it, look it up after the episode, but there there's a line. You hear a lot in the software dev world, which is write software for the poor bastards is going to have to maintain it two years from now because that poor bastard might actually be you. Right?
Speaker 1:Hmm.
Speaker 2:There's a lot to be said for thinking of a future in software development, which, which gets back to your original question, which is,
Speaker 1:But the one that the one I've heard is always write your code is if the person who's going to be maintaining it is a murderous sociopath who knows where you live.
Speaker 2:Okay. There, there is a, there's a, there's a flip side to that joke, which is debugging code. It's like being the detective and the murderer at the same time.
Speaker 1:Yes.
Speaker 2:But, um, but no, but point being that, that takes us back to I'll pull us out of this rabbit hole. Did I hit that? I'd done for us, but that takes us back to your original question, which is why companies build custom software. And so when I said that, you know, you write it once and it replaces so many people that that is an over simplified truth, there a big asterisk on the end. But as long as you realize we're going to build this software and the software itself never ends. But in theory, you know, the, the marginal cost of adding new customers or whatever, whatever we've heard cheaper than the marginal cost of adding, um, new employees to handle this manually, you know, that's where software really shines. But the flip side of that is you've now built something that you need to maintain forever. And, um, and it's not just having to add new features and fix bugs, something else that bites a lot of companies when they build a custom software, um, if they don't have a lot of in house experience, building custom software is the realization that software constantly changes. Even if your business requirements haven't changed the underlying operating system on which your code runs, eventually that's going to be what's called ELL or end of life, which is a nice way of saying that the manufacturers won't support it anymore. Right? So from just a pure IP policy perspective, you need to upgrade your operating system, which affect your code. You might need to change providers. You might need to do all sorts of things that have nothing to do with your underlying business model or what you're trying to opp or what you're trying to automate. But in the end, you're still going to have to change the code a lot. And just getting used to that, we'll say these companies of trouble and see if the developers doing the work out of trouble as well.
Speaker 1:Um, so, so bringing sort of like things back to the, the, the, the role of, of software developers, um, you know, um, you know, our R D do you think that software developed, like, regardless of what the, the, the companies like business model, uh, is, uh, whether they're doing off the shelf software or building custom software for themselves, for others, et cetera, um, our, our software developers just sort of, you know, cogs in the machine, like, you know, code churning like machines, or, or do they, or do they have the potential to like play other roles within the, uh, the company?
Speaker 2:Um, the answer to that question is yes to all of the above. Um, and by that, I mean, there are situations. And when I say situations, even a combination of the developer, him, or herself, and then the employer, uh, there are situations where it is entirely possible for a developer to spend their entire career just writing code. That's what they really want. Um, but as a software developer, you have so many other options within a company to move around and learn things. Um, in fact, going back to what we were talking about before this notion of the business of software, whether you're building software, uh, that will only be used inside your company, or whether you're building software for someone else. Um, think of all of the different people and roles involved, and when you're a software developer, if you so choose your work, can touch all those other roles. For example, as a software developer, could you sit in the basement and only look at the specs all day and write code? Could you also get up and talk with the product people talk with the salespeople, if you're lucky, maybe even get out and talk with the end users of your software? Yeah. In fact, I would highly encourage that because that makes you a better software developer, because you're getting to see how your software is being used and treated in the wild. Um, it also makes for a better software because you're going to get a chance to see how it's used and how you can predict what changes might come down the road. And you can account for those. And also sort of circling back around to your original question that opens the door for other roles you can take on in the company. I mean, one of the great things about the tech sector is that it's, I wouldn't say it's easy, but it's easier than in certain cases to move around. I know people who've gone from have started off as software developers, and it was great at that. And they eventually gotten floored. I want it to do something else. And they were working in a large and supportive enough company where they can say, okay, last year I was a software developer. You know, this year I formally joined the product team or this year I became a predict manager or I've moved in, or actually I know a couple of people who have even gone from actually hands on keyboard, writing software to joining the sales team and going out into selling it. Um, so it's a great way to spend time inside of a company, learning how it operates and then figure out what else you want to do, because most of the hops you would want to make. They're usually one or two hops away from being a software developer, which makes it a lot easier for you.
Speaker 1:Hmm. And I know that when, when you've, uh, when you've spoken to my class, you've, you've sort of described this as sort of the, the, the, the cheat codes of, uh, of making it in the, in the software industry, or are there, are there other like cheat codes that, uh, uh, that you think people should be aware of?
Speaker 2:You know, I'm, I'm, I'm glad you mentioned that the cheat codes, I would say, actually, I can take you to what was usually the last slide in that presentation, you know, which was the common thread and the common thread among all of those quote unquote cheat codes was get away from your desk and talk to people. Um, like I said, talk to the product team, talk to your stakeholders, talk to the sales crew. Anytime you can get out and just see who is using your software, who is selling it, why it's being built, it's better for you and better for them. So that's why I say I could rattle off the six cheat codes here, but instead I would just say anything, it'd be creative. Anything you can do with the developer to get away from your desk, go do it. Now, I'll definitely write, you want to do your job. If you're hired as a software developer, in the sense of while you're there to write some amount of software, to make sure you do that, but also make sure you accept opportunities. And quite frankly, even make some of your own opportunities to get away from your desk, maybe get outside of your team and see what else your company has to offer. And this is, this is true in any company, no matter how small, but it is especially useful in the larger companies, just because one of our secret of large companies is that they tend to be islands unto themselves, or if they tend to own the entire vertical, which is for the students that's industry, speak for owning every aspect of what the company needs to it's really self-sufficient right. For example, um, that's why it's many large companies have a sizeable internal software development effort. It's because they have enough development going on. They can afford to build out an entire department and so on and so forth. And what's, what's great about the larger companies has been because they're so large because they have 20 different departments. It's many different roles. Um, it's much easier for you to have several jobs within the same company than it is to leave and try to start fresh with a new type of role in a different company, right. You've already proven yourself in your original company.
Speaker 1:Okay. Well with that, I think we can wrap things up a Q a thank you so much for
Speaker 3:Talking with us today.
Speaker 2:No, and[inaudible] thank you. Once again, for inviting me, I realized this year is a bit different with the pandemic. I wasn't able to come down and visit the students in person, which is why it's fun to do that. But I'm really thankful that you're doing this podcast so that the students can still indirectly meet all of these industry professionals and learn some new tips. And as always, thank you once again, just for having this class, this is something I have not seen elsewhere. And I think the students that, that take this class of those winter industry, they're grilling, they're really going to have a leg up over kids who haven't eaten any of this before.
Speaker 3:Now. You're making me blush, which thankfully, since we're doing a podcast, you can't see, uh, anyway, that, that is it folks. Uh, bye. Goodbye.