AI in AppSec: Hype, Layoffs and What's Actually Real

Cole Cornford (0:00): I think it's all bullshit. I haven't seen many situations where AI has been the absolute root cause behind why somebody has had their job replaced. We've done that kind of stuff previously in the past with like FilterSet. Ultimately what I find is it just creates a hell of a lot of noise and very little signal for people as opposed to like manual interrogation and stuff. So look, jury's out. Let's see how it goes, you know? I'm Cole Cornford, founder and CEO of Galah Cyber, and you're listening to Secured, the podcast where I catch up with developers, security leaders, and innovators to talk about the real world of AppSec. Open source now powers over 90% of the software we build, but it's also where attackers increasingly strike. ChainGuard closes that trust gap with hardened, secure, production-ready open source builds so teams can build faster, stay compliant, and eliminate risk. Get your free CVE reduction report at dayone.fm/chainguard and start shipping software with confidence. Hey everybody, it's Cole. I've decided today to do a solo episode to talk about a couple of interesting things I've been seeing in the application security space. The first thing that I've really wanted to talk a bit about is that like there's just so much noise and craziness when people are just talking about the potential for artificial intelligence. And as someone who's been spending a lot of time either with companies seeking to secure artificial intelligence, but also companies that are looking at building products around artificial intelligence, I'm kind of uniquely positioned to talk a fair bit about where I'm seeing it being used and like where there's a lot of hype and marketing fluff and things are not going as well as they appear to be in the broader media, right? So the first thing I'll talk about is just the Claude Code Security piece. So like I'm recording this episode on Friday the 27th of February and about a week ago Claude Code Security was released. Yeah. For those who don't know, when you run, you have Claude, and Claude basically is a copilot that can sit in your IDE or used as an agent to go and produce code, and it is able to either suggest things or go off and build things for you reasonably well. Claude is the darling of many, many people, and so when Anthropic came out and said, well, now Claude has the ability to do secure code review as well, so you can write your code, and then you can run the, you know, the agent, or you can just ask Claude to make sure it's secure. One of the first things that happened is all the cybersecurity companies had a massive drop in valuation because they all saw Claude coming in, AI disrupting that whole field, right? To the point where a bunch of my friends who are building AppSec companies and product businesses are now struggling to be able to even raise seed funding or Series A funding. Big because of that announcement. So, and look, I understand, I actually think it's a good thing, like, 'cause we want to be releasing products that make it easier for us to secure our software. But there's, I don't think it's, it doesn't make as much sense to me to have the sheer, you know, market sentiment, like just because something, you know, doesn't. meet the goal doesn't mean it can't be exceptionally harmful to talk through it, right? One example is if you are, you know, writing code, the first thing that you're going to say is like, well, why, why wouldn't Claude ask this, write secure code by default? Why do I need to have something in an adversarial relationship with it? And I mean, that's a pretty interesting point. There's actually many circumstances where security is not the objective, it's not the end goal. And while we try to do what we can to write secure by default code. Like, it's, it's not always achievable if we're trying to just meet different business objectives, right? That context really matters. And so something that when you write code locally and then get an agent to look at it, the context that it has is the static code, like the, what, what's available within the repository seeded with information to give us some context potentially about your company or about the products that you're building, or maybe the types of systems it's interacting with, or what it can ascertain from going off and talking to its own few other endpoints and stuff. But that's kind of where it falls over, is when you want to talk about security issues, the vast majority of meaningful and impactful security issues are not because somebody, you know, left a parameterized query, or did a, had a hardcoded credential, or start used a— Yeah. A thing like dangerously set innerHTML. Like those do occur, but they're found quite easily with existing security tools. The meaningful problems are always when somebody creates some, like puts something in and makes assumptions about how it operates in the broader environment. And it's usually a chain of a lot of different bits and pieces in some kind of production ecosystem that leads to some problem, right? And it's why penetration testing has never gone away. Because people have so little confidence that these tools are able to get context about your business and about how that application is working. So I don't really see Claude Code doing as much as people seem to be thinking. The other thing is like, we already have a lot of existing tools to manage and, and like identify security vulnerabilities. Like the biggest problem has never been in like finding bugs. We're really good at finding bugs. We've been finding bugs since the '90s. They're just either manual interrogation of source code or like asking like tools like FindSecBugs or Fortify back then to just statically look at code and just find bad patterns or traverse the AST, right? Finding things is never the issue. It's like choosing what things to fix and making sure people understand why they're fixing it and making it hard to make us reintroduce those things in the future. Like, I don't want to be spending a bunch of tokens on, you know, bumping dependencies or fixing, like, you know, secrets that are not relevant because they've been rotated already, or like having to update configuration of specific assets when it's configured that way for a specific reason. I just see AutoFix doesn't really work as well as you'd think because you're going to, like, solve security problems but create reliability ones, right? Lastly, like there's a question about, is everyone even using Claude Code? In my experience, like the vast fucking majority of people that I speak to, like kind of use it as a bit of a hobbyist thing, but I've seen almost no organizations except for really small places, like say, yep, everybody goes and uses this and they have to use it. And if they're not using it, they're going to get performance managed and pushed out. Like there's a couple of places doing that, but like 99% of institutions have a couple of hobbyists who use it very well and a couple of people who are using it incredibly poorly. And the vast majority of people don't care which way or other and just doing what they've always been doing. And so if you are telling me that you are only going to target like that 1 to 5% of like elite power users who really want to do the best thing possible, then, but that's also the group of people who are the least likely to introduce security vulnerabilities because they're the best at their fucking jobs. So I'm not too worried about this replacing AppSec as people seem to think. I think it's actually kind of embarrassing. It shows a distinct lack of maturity on people if they think it's going to do that. Anyway, the next thing I wanted to talk through was like the Aikido Infinite release, which came out yesterday. And I see almost exactly the same kind of things coming up for that, which is they've effectively said, any type of cybersecurity professional service, we're going to try to move it to happening on a continuous basis because everything else is a point-in-time assessment. And what we can do is just do continuous assessment of like your code, your, um, your assets, your, um, do penetration testing of continuous reports and, and help you like, you know, test things, catch things, and patch things, right? I really worry about Again, a few aspects here. One, the vast majority of software assets are just not going to be operating in an environment you can be doing that in. They're not SaaS businesses. They're like, if you set it up in like maybe an integration environment to be, you know, continuously testing and finding things and so on, like great. But like most places don't have an architecture where it's you know, production has parity with dev and test and staging. And I also— there's many production assets that are just not going to be like exposed or accessible on the internet, like OT infrastructure, like citizen-developed software applications on people's workstations. Like, I struggle to see how this has much market penetration outside of like hitting all of these high-level things. I, um, I worry about cost. It's expensive to be running things on Bedrock, like $2,000 to $3,000 a month or more. So if you're willing to pay that for like continuous scanning, wouldn't it be better to just like look at hosting your own server or running, like running your own models and stuff? I just worry about like the ability for you to be able to say that this is worthwhile in and good to do. Again, resilience is a huge conversation here. Like, do you want to have, like, AI agents continuously penetration testing things? You don't necessarily know what the outcome is, and if it's going to break production, you're in a bit of trouble there, right? So, and then it goes back to, I'll replace human pentesters or human analysts. Like, the novelty— like, we've been training these things on, like, clearly defined patterns, and while they do come up with occasionally novel things, most of the time any AI pen testing tool I've seen, or like just AI-related security product, uses existing patterns to go ahead and like find stuff, right? We have a massive backlog of things that we need to, you know, identify and, you know, patch. So I think it'll be really effective and helpful for that. But then as adversaries start to use these things to create novel and sophisticated techniques to break into organizations, Like, I don't see how that's going to become training data for these AI pen testing systems or being remotely representative of the stuff that's happening in them. So, like, I think it's cool. I think it's a cool idea. I like the idea of continuously assuring your environment. I mean, daily scans or nightly scans or continuous scans have always been something that we've looked at, as opposed to just running scans in a DevSecOps pipeline. But I do worry about the, what would you say, noise is probably the other thing I'd be really concerned about. Because even if you have AI systems that are triaging these things and just saying like, oh, we have high confidence in these vulnerabilities and low confidence in these other ones, we've done that kind of stuff previously in the past with like filter sets or like, you know, like trying to say, hey, are these reachable and exploitable, or looking at using ASPM to combine different types of tool findings together. Ultimately what I find is it just creates a hell of a lot of noise and very little signal for people, as opposed to like manual interrogation and stuff. So, um, look, jury's out. Let's, let's see how it goes, you know. So, and I wish Aikido the best of luck. The next one and last one I want to talk through was about the whole AI taking jobs narrative. Um, I think it's all Bullshit. I haven't seen many situations, like many at all, where AI has been the absolute root cause behind why somebody has had their job replaced. Like, yeah, I guess you could probably say like, look, if there's like, you know, you need 20 people to do something and then, you know, now you only need 15 people because AI's automated that stuff. I think what you'll find in this, in not too distant future that that'll probably scale back up to 20 to have to deal with the editing and reviewing and the changing of whatever stuff the AI is doing, or there'll be new jobs created around managing and governing this. So it's kind of a shift of jobs. I don't see it as a reduction. What I really see it as is actually a smokescreen to hide from just having, you know, a lot of scared investors doing stupid things. You know, you have, let's say the two recent announcements were WiseTech. and Block, and all of the coverage from both of them has been they've cut thousands of developers and customer service agents, and it's going to be terrible because AI is taking over development. We don't need more developers. And we've seen this at Klarna in the past, and I've seen it recently with Woolworths in Australia where they had to go bring like customer service back basically because people couldn't get what they wanted out of the bot and they were leaving that business. And with Woolworths, they had a, an agent that was telling them about their, their supposed mother when they're a robot. Like, it doesn't make any sense to me, you know? So yeah, I speak— it's going back to those two initial companies. Um, I think that they both had terrible business performance this year. Like, WiseTech is a logistics firm. The founder of WiseTech has had a lot of controversy around, uh, making some decisions that are not very ethically minded. It's up to him to choose what he wants to do with his life, but it's had a significant impact on the governance of his company. And if they're too busy dealing with like constant PR problems, like you're going to struggle to attract good talent, you're going to struggle to be focusing on your business, right? So that, and very recently they had the whole SaaS sell-off, like most SaaS businesses in our economy like whether they were in America or Australia, like got really, really heavily sold off by retirement funds, by investors, et cetera, because they're feeling that they're just maybe not replicable, but just feeling that they're a little bit too expensive for how much value that they really provide. So, and we saw that with Atlassian. Atlassian's like 60% down year to date in like that, that's tremendous wealth deprecation. And like, I'm confident in the next like month or two we'll be hearing about Atlassian making redundancies from artificial intelligence as well. Right. But yeah, we even go to Block. What, what's Block done? Block, old mate at Block was very into cryptocurrency and he bought a metric fuck ton of Bitcoin. If you got like, you know, a billion dollars of Bitcoin and then that billion dollars suddenly becomes, I don't know, half a million dollars, then I would be very concerned. Yeah. That's a lot of wealth to lose. And so I, I don't think that these AI layoffs are really as much as people seem to think they are. I just think that there's private businesses that are not doing as well as they should be, that have been mismanaged and then are using it as a reason to be getting rid of people and trying to give a good news story to the market. And there's lots of like, you know, people theorizing that like there's, you know, economic headwinds and inflation and capital debt and all of that. And I, I'm not sophisticated about any of those kind of concepts. I just think that it's just people trying to manage the messaging and using this as an easy cloud cover and smokescreen for doing something that they've wanted to do for a long time, which is make themselves more profitable. And get rid of headcount in a way that's not going to tank the share price. So, but anyway, those are the 3 topics I wanted to talk a little bit about today. If this format's interesting for you, um, let me know. I'd love to maybe start doing this on a weekly basis in conjunction with my normal interviews and just talk about my thoughts to do with some artificial intelligence, or I guess software security concepts. Anyway, thank you all for coming on, and I'll see you next time. Thanks a lot for listening to this episode of Secured. If you've got any feedback at all, feel free to hit us up and let us know. If you'd like to learn more about how Galah Cyber can help keep your business secured, go to galahcyber.com.au. If you're new to the show or a seasoned veteran, I want to hear from you. I'm always looking to do better as a podcast host and as a person. So tell me about what I can do better. Tell me about the types of people you want on on my podcast. Tell me about if the format's working for you or styles need to be switched up. Do I need to interview differently? So on and so forth. I appreciate you sticking with me for almost 30 episodes at this stage, and I hope I can keep learning and you can keep learning and being entertained for many more episodes to come. So please shoot me feedback. I want the good, the bad, and the ugly to pod@glasscyber.com.au. And if you don't like email, hit me up on LinkedIn. Quick update, I'm launching a new course for 2025. It's Foundations of Application Security. It's a 2-day bootcamp covering everything you need to become a well-rounded AppSec practitioner. There's both public and private courses available, and the best part about it all is taught by me. So if you're interested, just send me an email, col@galaxycyber.com.au, or say hi on LinkedIn if you can have a chat about it. Hello.