Silicon Futures for November - SC 2024, Qualcomm Investor Day and more! (with Karl Freund and Jim McGregor)

Jim McGregor of TIRIAS Research and Karl Freund of Cambrian-AI Research joined me to recap another action-packed month of November 2024 in the world of semiconductors and accelerated and non-accelerated computing on the neXt Curve reThink Podcast series, Silicon Futures.

We parse through the key announcements and semiconductor headlines that mattered in November October:

➡️ November, month two of the crazy season for industry analysts (2:04)
➡️ SC (Supercomputing) 2024 - AMD taking the spotlight with El Captain (3:39)
➡️ Scaling laws in GenAI supercomputing and GenAI (6:00)
➡️ Qualcomm's Investor Day 2024 - Is diversification working? (11:27)
➡️ More takes from SC 2024 - NVIDIA's Moat (25:35)
➡️ The GenAI supercomputing hype residual (27:16)
➡️ AI-RAN - GenAI supercomputing at the edge (27:40)  
➡️ How will AI change your company? Jim drops some wisdom (29:01)
➡️ Getting beyond crap AI to good AI (28:07)
➡️ The heterogenous compute blob that is accelerated computing (34:21)
➡️ 2025 - The year of GenAI what? The experts chime in (38:39)

Hit both Leonard, Jim, and Karl up on LinkedIn and take part in their industry and tech insights. 

Check out Jim and his research at Tirias Research at www.tiriasresearch.com.
Check out Karl and his research at Cambrian AI Research LLC at www.cambrian-ai.com.

Please subscribe to our podcast which will be featured on the neXt Curve YouTube Channel. Check out the audio version on BuzzSprout or find us on your favorite Podcast platform.  

Also, subscribe to the neXt Curve research portal at www.next-curve.com for the tech and industry insights that matter.

Transcript

[NOTE: This transcript is AI-generated and may contain inaccuracies and misrepresentations.]

XC Webcast 2024-Silicon Futures November

Leonard Lee: [00:00:00] Hey, welcome to this next curve. Rethink podcast episode where we break down the latest tech and industry events and incidents and, uh happenings into the insights that matter. And I'm Leonard Lee, executive analyst at next curve. And in this Silicon futures episode, we will be talking about.

Leonard Lee: A lot of stuff, mainly the month of November and all the exciting things that have happened in the world, the semiconductors, and I am joined by my wonderfully fabulous friends. In the industry analyst community, two individuals who I regard very highly, Carl Freund of Cambrian dash or hyphen AI research and, the equally fantabulous Jim McGregor of the famed Tyrius Research Group.

Leonard Lee: And, uh, how are you gentlemen doing? It feels like an eternity. Still, 

Jim McGregor: I'm rethinking your, I'm rethinking your [00:01:00] terminology now. . Oh really? Fantabulous. . 

Karl Freund: I have the best words. Fantabulous is a great word. Yeah. . 

Leonard Lee: I have the best words. So how are you gentlemen? Doing? Doing 

Karl Freund: good. 

Leonard Lee: Doing 

Karl Freund: good, doing good. Happy to be here, buddy.

Leonard Lee: Yeah. Okay, awesome. So before we get started, remember to like, share and comment on this episode and also follow us@uhwww.next curve.com. Also, you'll find us. On YouTube and Buzzsprout, here every month. So I'm doing this with Carl and Jim, and we're just trying to bring the latest, greatest, and hottest topics in the semiconductor industry.

Leonard Lee: And so join us every month and you will. You will be in the know. So, uh, with that, uh, November crazy month, I mean, I think almost every single month is a, is a crazy month in the world, the semis. , what do you guys think? , what did you [00:02:00] intend first off? Because we're all, this is like the high season for events.

Leonard Lee: But maybe Jim, let's start with you. 

Jim McGregor: I didn't intend as many things as I expected because I was ill, but my colleagues did. So, I mean, there was a lot of things from the end of October through November from OCP to SC24 IBM's Quantum Developer Conference Microsoft Ignite.  on and on and on, just so many things and so many announcements.

Jim McGregor: And,  and a lot of announcements, not just about semiconductors, but really about systems and systems architecture as well. So really impressive, even EDA tools,  seeing announcements by companies like Siemens about their new cross the board UI that uses AI. So there were, there were tons of announcements to discuss.

Leonard Lee: Yeah, so any particular ones that you want to highlight or do we pass the ball on to Carl and give him a chance to 

Jim McGregor: Let's [00:03:00] let's go.  the first thing I'd like to highlight. Well, no, let's go into Carl and then let's go We have so many things to discuss. 

Karl Freund: Like you Jim, I got a little bit ill this month But other than that, I was able to attend a few events.

Karl Freund: He virtually in person I think your comment about systems is is sort of the highlight of the month If you look at, obviously Supercomputing is all about systems. Nobody cares about the chips. Well, they care what they buy as systems. Yeah. Obviously,  a MD, , was riding high from their, uh, turning on, , El Capitan.

Karl Freund: And That's right. And that was a big deal. It was a big deal. I,  what I, when I think about that, it's like, well, how did, how did in Nvidia drop the ball there? And I was at AMD while those chips were being designed. The focus was really on winning DOE's love and attention and money. And, , therefore they really focus on 64 bit float.

Karl Freund: And yeah, it has decent AI performance [00:04:00] too. But their priorities were HPC, HPC, and then AI. And you look at NVIDIA, it was, their focus was AI, AI, and AI. for your time. And so as a result, you're seeing the demonstration of the, prowess of A MD and HPC, but also the prowess , of Nvidia really cementing their lead in ai.

Karl Freund: I'm not hearing a lot of people talking about MI three hundreds anymore. I'm sure they are. They're not talking to me about it though.  and with the focus really being on Blackwell. So I think it's, it's not that one company screwed up and the other one didn't. It's really, they had different strategies and you're seeing that playing out on their Silicon and we'll see where that goes with the roadmaps.

Jim McGregor: Well, and also, especially for the DOE and these labs like,  Frontier and El Capitan, these supercomputers, they were focused on the system, they were focused on a massive supercomputer. And. And to that point,  AMD was [00:05:00] offering the complete system solutions. They were offering the processors, the accelerators, the, the DPUs, they were offering all that, and it really hasn't been till recently that we saw that come from NVIDIA, especially with, I mean, Especially with Grace.

Jim McGregor: Grace is now, proving its worth. And we, we, we didn't, we've seen a lot of system innovation from them, but just recently, and that's through the GB200 with the Blackwell generation. So yeah, I, I would agree with you. And if people have to remember that a lot of these systems, just like semiconductors, they were planned for.

Jim McGregor: Four or five years ago. So,  a lot of decisions were made years and years ago. That's why they're not even on some of the latest technology at this point in time. 

Leonard Lee: Yeah, and,  the other interesting topic that keeps coming up lately are these these discussions around scaling laws. We're hearing it all across the board.

Leonard Lee: And 1 of the interesting things that I'm noting is, as we start to see these comparisons across the HPC [00:06:00] universe. Like what you're talking about, Carl, HPC versus quote, unquote, AI supercomputing it,  the, the data type right in one corner, you have HPC where, like you mentioned, you, you have 64 bit float and then 64 bit and then you have , what you have.

Leonard Lee: And video talking about quite a bit and the rest of the generative AI super computer versus the four bit. Yeah. And, but they, they try to compare each other using these two different, data types. Right. And it's not a fair comparison, right? It really isn't, but then. It's causing a lot of confusion.

Leonard Lee: I think in the marketplace where you have one side claiming, Oh, we have leadership and supercomputers like, well, wait a minute. Well, if we, we apply 64 over here, it's, you're not scaling as quickly as, uh, might be asserting, but there's now [00:07:00] increasing confusion around scale and whatever people are referring to as scaling laws.

Leonard Lee: But that's something that I noticed in a lot of the events that I've been attending where you're hearing a lot about scaling. I mean, it's literally prefaced in many of the keynotes that we're hearing and most recently at Microsoft ignite. So yeah, but this is a, I think this is going to be kind of like a semantic shake.

Leonard Lee: Down, that's going to happen over the next 6 months, right? As  I think there's going to be a system level bid for diversity, right? , making these system investments more multipurpose. So let's say not just for AI, but for all the other kind of. Supercomputing, especially off of this idea of general purpose, , accelerated computing.

Jim McGregor: Yeah, well, NVIDIA is trying to do that, obviously, with supporting not just AI workloads, but also RAN workloads. So for the wireless [00:08:00] networks, and obviously they, they have an announcement there with SoftBank. So I would agree with that. I would also agree that,  and we've seen this from Jensen. We've seen this from, uh, Pat.

Jim McGregor:  uh, Gelsinger, uh, say that,  the new unit of compute is the data center. And really the biggest thing is, is you have to start building these solutions at the building level. And then, and then going all the way down through the infrastructure, through the network, through the chip sets or through the solutions you select,  and we're seeing this,  now we have AMD and NVIDIA or AMD and Intel offering.

Jim McGregor: different types of CPUs. So you may use different types of CPUs in the data center. You may use different types of accelerators, one for inference, one for, uh, uh, training, one for HPC,  and it, and definitely, uh, DPU for offloading some of the overhead workloads.  and networking,  networking is becoming so critical,  to be able to handle [00:09:00] this, not to mention the memory bandwidth.

Jim McGregor: So it really becomes a, Especially for the data center, it becomes a building level challenge. 

Karl Freund: Yeah, I think this is, you're exactly right. And this is why we see NVIDIA putting more and more emphasis on the networking, whether it's NVLink or InfiniBand or their version of Ethernet. Uh, they're, they're focusing a lot of attention on that because that's the bottleneck now,  And I think that,  the, the, the power constraints is the next bottleneck.

Karl Freund: And then you get to memory and then you get to the actual performance of the chips. And so when you think about it from a data center level, it kind of flips everything on its head as to what the priorities are and new players will emerge. Uh, whether they're,  doing,  new networking topologies, like one of my favorite startups I'm interested in called, uh, in Fabrica, uh, which is doing for the back end, what NVLink does the front end.

Karl Freund: So I think we're going to see a real shift because of [00:10:00] what you're talking about, Jen, as, as, as everybody gets more and more focused on data center scale. 

Leonard Lee: I don't know why I put on, no, do not disturb. Anyway, 

Jim McGregor: that's okay. We disturbed you 

Leonard Lee: Yeah, Apple intelligence.

Leonard Lee: There's one thing I 

Karl Freund: haven't found it yet. My whole room is full of Macs here and I don't, I haven't found Apple intelligence yet. 

Leonard Lee: So which was, which was one of the big, releases of November or at least the tail end of October. And yeah, I mean, I have to really be honest here. I'm, I'm not all that impressed, but then,  we have to remember they call it beta.

Leonard Lee: Right. That is true. Hey, this is stuff is ready. I mean, officially, we're supposed to treat this like a beta, right? And so that there's going to be limitations and they're doing this progressive rollout of features, which is probably smart, quite honestly, but  I, I'm just not finding it as useful [00:11:00] and helpful as just, just like co pilot no comment.

Leonard Lee: Yeah. So,  Yeah, so what,  I know a couple of things that I attended Qualcomm investor day I was there. I know you generally weren't able to make it, unfortunately, because you're ill, but I thought they're the key takes there for me were this push. Toward the growth markets, right?

Leonard Lee: And it literally emphasized only that they didn't really talk about mobile much at all. QTL almost no mention. It was all about automotive, where they continue to make traction, especially with connectivity and. Most prominently infotainment and they're making a bid toward,  really tapping into this whole ADAS market, not autonomous, 

Karl Freund: which 

Leonard Lee: I always thought was really [00:12:00] smart because.

Leonard Lee:  with the autonomous stuff, it's niche and your, your sites are too far out in the horizon. We've seen what's happened with a lot of companies that have been orienting themselves toward a, uh, more of the, let's say, SAA. SAE level four to five they've had to throw all this stuff back toward,  three and two and a half, right?

Leonard Lee: So 

Karl Freund: 3NT seems to solve a lot of the customer's needs right now. I'm not sure customers are really ready to trust four and five. So I think that what they're doing is very smart. In fact, if I could just predict, I would say they'll overachieve on their automotive. The one I'm more worried about , , is the PC, uh, where, yeah, they, they, they've got a good chip, but that's a tough ecosystem to crank into and produce.

Karl Freund: I think it's what, 4 billion in revenue. 

Leonard Lee: Yeah. 

Jim McGregor: Yeah. Which is probably 10 to 12 percent of the market by 2029. [00:13:00]

Karl Freund: Yeah. 

Jim McGregor: That's a lot. Yeah.  and I, I agree with you. I think they, they have a great, uh, strategy for automotive,  having being the dominant player in telematics, being a, becoming a dominant player in infotainment , definitely you'd be foolish not to look at them for the command and control systems for the ADAS systems if you're already using them throughout other parts of the vehicle.

Jim McGregor: Yeah, absolutely. 

Karl Freund: How do you think that'll shake out? Do you think that they'll end up owning the bulk, or if anybody owns the bulk, of being a market leader in infotainment and somebody like NVIDIA will, will take care of the heavy lifting of ADAS? Do you think?  

Jim McGregor: it, the funny thing is the OEMs from what we've talked, the ones we've talked to, they're all hedging their bets.

Jim McGregor: They're working with multiple partners. They're working on their own solutions.  matter of fact, if you go to a tier one like Conti, Continental they'll say, okay, here's our command and control box. What would you like in there? Would you like a Qualcomm board? Would you like an NVIDIA board?

Jim McGregor: Would you like, you [00:14:00] know, same box? So even they're hedging their bets but no, I think it will, a lot of it's going to come down to the software and really having a, because we keep calling it the software defined vehicle and having that integrated software solution, but still being able to customize and control it because nobody wants to give up control of the vehicle to Apple or Google.

Jim McGregor: So I think having that software, that unified software platform is going to be critical.  they, they don't have the training solution. So if you're going to train today, you still got to go use Nvidia. But,  I think having that unified software solution is going to give Qualcomm an edge up. 

Leonard Lee: Yeah.

Leonard Lee: And, but I think that's where this whole, and they didn't talk about it a lot, but the whole one technology roadmap strategy is important. And I think AI hub is taking on a more prominent role. In fostering that strategy on the front. Right. And so, as you look at them, take, like, for instance, not cool.

Leonard Lee: Always talks about chassis. He [00:15:00] loves chassis. Right now, taking that to. The industrial IOT, I mean, the whole idea is how do we make all of these assets that we have that are optimized for the Qualcomm stack and port them over into different domains and growth market and which end up being the growth markets that they're targeting.

Leonard Lee: How do we create that technological synergy and portability that brings the optimized goodness that they have? And, , I think they have a great starting point with automotive just because they've had to build safety critical systems and they have. Now, an understanding of that, porting it over into industrial IoT, which is the other big sector that they want to diversify into, which they're bringing a chassis concept.

Leonard Lee: There is going to be it's going to be interesting because industrial IoT is such a messy. 

Jim McGregor: But let's face it guys, if, if the U S auto, or I should say the [00:16:00] Western auto manufacturers can't figure out a way to produce cars more economically they're all going to lose to China. So it's not going to matter.

Karl Freund: Yeah, absolutely. And, and China seems to know how to build electric vehicles too, don't they? 

Jim McGregor: Yeah. Well, when you're subsidizing,  30 to 50 percent of the cost, 

Karl Freund: it's 

Leonard Lee: a little bit 

Jim McGregor: easier.

Leonard Lee: But  we also have to get off of that because at the end of the day, we need to,  like the Western companies need to figure out how to compete because we've seen this, we've seen this play out before and never,   If you focus too much on what you might deem uncompetitive, it detracts you from actually innovating and figuring out a better way of competing.

Leonard Lee: And that's what frustrates me when we have people,  focus too much on, oh, well, . [00:17:00] They're subsidizing XYZ. It's like, no, no, but how are you going to figure your stuff out? Because,  the chips act, that was the other thing that it's a big news item. Just, I think it was today is there's this rumor or report from Reuters that Intel might be getting less love from the chips act than originally anticipated, but  focus on subsidies here in the United States, rather than having that solid plan of competing without the subsidies, which which end up being much smaller in terms of , the investment that's required how has that benefited The industry, 

Jim McGregor:  it worked, it worked out so well for the solar industry.

Leonard Lee: There you go. But Hey,  what? I'm, I'm really curious. So S C 2024, I wasn't there. I didn't even listen in on any of the stuff. So any, any cool takeaways or highlights from that [00:18:00]event?

Jim McGregor: There were a lot of announcements, obviously, about new supercomputers and everything else. AMD taking kind of the lead spot on the top 100.

Jim McGregor: And  also taking the lead spot,  now going from Frontier to El Capitan. And there were other announcements. NVIDIA made a bunch of announcements around some of its new NIMS and models that are available and some of the solutions.  and quite honestly anybody that thinks NVIDIA's moat is just its hardware is foolish because their software strategy is really their strength at this point.

Karl Freund: Yeah, I think people, people fall into the trap that there's, there's their moat is CUDA. It's like, no, no, no, that's, that's the bottom of the boat, right? And all the water on top of that is all the other software they're putting on top of CUDA, which drags CUDA along but I'm reminded of a, Conference I was at recently where a second tier cloud provider basically said that  their man actually the first tier cloud provider.

Karl Freund: He was now in a second tier. So I won't tell you who it [00:19:00] was, but they are saying their management came and said you had to start using our chip. And, and the team said, fine, we won't make schedule. We won't deliver on our performance goals. Cause we've got to spend all our time on the software. 

Yeah. 

Karl Freund: And so their software mode is deep and wide and, is not just CUDA.

Karl Freund: If people use the word CUDA, that that's, that's not it. 

Leonard Lee: Yeah, you're right. I mean, that is a much lower level. 

Jim McGregor: And I think SC24 was kind of a critical point,  we're seeing a lot of transition over towards different forms of liquid cooling. I'm still not a big fan of immersion.

Jim McGregor: I did that in military applications. I think you're, I can tell you all the problems with it. But no, also,  we're on the cusp of having UA link,  a UA link solution come out next year. You Universal Ethernet specifications should be completed shortly or early next year.

Jim McGregor: So we're going to have a lot of new networking. You're expecting UA Link 

Karl Freund: to ship? You're expecting UA Link to ship this in 2025? Well, 

Jim McGregor: actually [00:20:00] AMD's Pallara Actually supports supposedly supports the 1. 0 specification for the universal Ethernet consortium. So UEC, so yes, so a lot of that technology is coming out very, very quickly.

Jim McGregor: So, I mean, seeing the, the investment in the infrastructure, the investment in the networking,  and continued investment just in supporting,  uh, scaling. As you pointed out is a critical issue, scaling solutions going forward and making more economical. 

Leonard Lee: Whatever scaling means these days, it seems like everyone keeps pivoting around all over the place.

Leonard Lee: But,  that's actually probably a good topic of research and study and clarify what the hell do you mean by scaling, ?  but  I want to go back to the comment around it's interesting. Just a 

Jim McGregor: quick comment for you. Uh, one of the things I've been looking at is,  how do we make the entire thing more efficient?

Jim McGregor: [00:21:00] Uh, because we tend to think, oh, well, we need to make the chips more efficient, or we need to make cooling more efficient.  I've actually sat down with, uh, NVIDIA, Vertiv, and a couple of other companies, and they've And the list I have is phenomenal. So, uh, it makes me very positive that we can achieve these, not just a hundred X, but thousand X jumps in efficiency from generation to generation.

Karl Freund: Yeah.  I think the information we publish is they're concerned about, or they're reported, reported that, uh, people can't install black well because of heating, it's overheating and so forth. To me, the, the, the point people should take away from that is. Is it this stuff is not easy and you need to pick the right partners because if you pick a wrong cooling partner, yeah, of course, it's going to overheat the wrong cooling solution, right?

Karl Freund: But if you pick the right cooling solution,  Microsoft and meta, they're standing up black. Well, just fine. Thank you very [00:22:00] much. Yeah. So I think that system focus. has got to be the theme, I think, for the next few years. 

Leonard Lee: Yeah, and it's also important to, to,  identify now NVIDIA and,  now probably increasingly AMD, that they're not just chip companies anymore.

Leonard Lee:  I mean, we've talked about it at this Yeah, 

Karl Freund: AMD's got already, 

Leonard Lee: previously on, on our, our podcast here, but, They're, they're building out like entire systems and you're making a great point Carl when you're delivering systems and you're working partners to just get,  customers to that,  the last half mile. There can be variances. That's where you, you're going to see the variances and the implementations and you could be engine,  a partner. That's not that great. Could be engineering problems into that system. It has nothing to do with the chip. Right? So. Chip is like 

Jim McGregor: way 

down 

Jim McGregor: there. That's actually why I think Verdiv is one of the most [00:23:00] interesting companies out there right now, because they, they supply that complete infrastructure to the data center.

Jim McGregor: They've got the electrical solutions. They've got the cooling solutions. They've got the networking solutions. That's almost what you have to have. And obviously we're seeing super micro kind of go that way with their new group focused on data center design. That's almost what you have to do to be able to compete in this market, to be able to service,  especially when you're dealing with data centers that were built,  5, 10, 15 years ago, , how are you going to rearchitect those data?

Jim McGregor: Those reuse those facilities, you have to have a partner that knows how to do that. 

Leonard Lee: Yeah. And,  I just want to throw a comment out there regarding NIMS. I. I think that's more of a quote unquote enterprise AI type of artifact. That's all about something that really related to supercomputing. So it's interesting that you get, you mentioned that because.

Leonard Lee: If I was NVIDIA, I wouldn't really focus on that as much as I would the [00:24:00]system and then the scaling part, right? Scaling these systems to whatever a million GPUs in the next generation and  why that matters and 

Karl Freund: how 

Leonard Lee: that's going to happen. I think they're focused on both. Yeah. 

Karl Freund: I mean, Jeff said,  we're, Hopper's ending at a hundred thousand, Blackwell's starting at a hundred thousand.

Yeah, yeah, yeah. 

Karl Freund: Nodes, right? And I think that the cooling solutions are getting pretty creative, right? I don't know if you've seen this uh, Flex acquired a company called  JetCool. And so instead of running fluid across the chip like this, they're actually, uh, injecting fluid down closer to the chip, it's much more cost effective, much more effective in cooling.

Karl Freund: And so you're gonna see a lot of more innovation in that space as, as people, people are not going to do is just, just say, well, I'll just buy air cooled for the. The foreseeable future, everybody needs to be preparing for liquid cooling. And it's 

Jim McGregor: funny you say that, because I also think that that's going to force a change [00:25:00] in system design.

Jim McGregor:  pretty much all the systems we use today are trays, or pizza boxes, if you've been around long enough to hear that term. And  and  when I was at Motorola, we did blade solutions. We did VME, Compact PCI, Best TCA. Exactly, but I mean they were extremely dense. We even put fiber in the back planes for some of the high end systems They were really really expensive, but they were really really good i'm just saying,  I think that that's going to drive a whole innovation and i've talked to some of these companies and saying hey have you taken another look at blades because working to a point where you need to think about the cooling and the power and everything else and i'm looking at blackwell and I'm, like,  I could have done that with blade systems, so much more eloquently,  without having 5, 000 cables going back up and down the backplane as a backplane for it.

Jim McGregor: But,  it's going to be interesting because I think it's going to drive a lot of innovation. 

Karl Freund: Well, those 5, 000 cables, 5, 000 cables brings up another [00:26:00] topic we should probably explore in a future a future broadcast like this. They're doing 5, 000 cables, uh, for the, you need the point to point connectivity to deliver the performance.

Karl Freund: NVIDIA intends to deliver with Blackwell. That's all going to switch to optical. It's a question of when. 

Jim McGregor: Yes. 

Karl Freund: And they talk about, well, we can't do it now. Gotcha. You can't do it. It's not cost effective. That's going to change. That's going to change in the next three 

Jim McGregor: years. And once again, we did optical backplanes 20 years ago at Motorola.

Jim McGregor: It's feasible. 

Leonard Lee: Yeah, it boils down to economics. Right. It is. I mean, that that's, that's been sort of the, the gating factor. And,  yeah,  one of the things that I think is becoming quite clear is, whether or not you think Gen AI has legs or not, it probably does is it probably, it may be shorter, maybe longer than what we think.[00:27:00]

Leonard Lee: But one, one of the things that's quite evident is all these innovations in supercomputing are, are gonna be the things that stick, right. And how they, what I'm really, starting to see is how some of the paradigms that are happening up here are starting to trickle back down. And as people are starting to look at,  edge deployments.

Leonard Lee: And so Jim, you mentioned, uh, SoftBank and NVIDIA collaborating or having this, actually, it's a new offering and architecture around AI ran. That's a great example. I'm not in the camp that thinks that all of a sudden you throw a GPU out there, you can have like a GPU as a service business and it's all,  net revenue.

Leonard Lee: It's not ran is cost and your GPUs out there are cost monetization. That is a separate topic, and that is one of the things that have undermined a lot of conversations around emerging technologies and their potential for delivering economic value. The thing is, is [00:28:00] they're separate. The GPUs are cost as well.

Leonard Lee: And whatever you use. to provide a service that is monetizable. That's still cost and you still have to go through a profit equation, part of the equation. But yeah, it's exciting though, to see how there's this influence coming down from generative AI supercomputing and all the innovations. That are now, I think, finding better economic profiles and that's something I think we should probably keep an eye out on as we're covering the industry because I have friggin funny feeling a lot of the stuff is going to start showing up in on prem and in certain edge environments.

Leonard Lee: As well as,  stuff from the bottom up as well. A lot of it originating from mobile computing, right? 

Jim McGregor: Well, I always have three questions for companies. First question is, is how is AI going [00:29:00] to change your company operationally? Second one is how is AI going to change your products and services?

Leonard Lee: Yeah. 

Jim McGregor: The third one, and not many companies can answer this one, is how is AAA going to change your business model? And what does your company look like in five to ten years? And if they say it doesn't, I worry. As a matter of fact, I asked that question directly of Jensen at GTC, and he looked at me and he says Jim, I wish I knew.

Jim McGregor: I love that answer. 

Leonard Lee: Yeah, yeah. Well,  it's interesting. I'm working with a couple of clients where I'm telling them,  I don't know or we'll find out is not a good enough answer. It really isn't. 

Jim McGregor: It's not, but you have to, you have to at least realize that AI is going to change all three of those aspects.

Jim McGregor: And if it isn't, you're not thinking big enough. 

Leonard Lee: Well, yeah. Yeah. Or you're not thinking clear enough as well. I, I know [00:30:00] I ask this all the time because I, whether it's 5g it's IOT or whatever, and to your point, Jim, a lot of times you hear,  Well, people will figure it out. I was like, no, no, no.

Leonard Lee: People will not figure it out. 

Karl Freund: Well, I mean, your competitors are going to figure it out before you. And,  I think, I think a perfect example of what you're talking about there, Jim is enterprise software market. 

Karl Freund: the impact of chatbots on enterprise software is not much. The impact for AI agents on enterprise software is massive.

Karl Freund: You can build the functionality you need. The AI. Agents can build the functionality you need using common software that's on everybody's Macs and PCs. 

Yeah. 

Karl Freund: And that's going to change enterprise markets. So,  everybody from SAP to Snowflake's got to, got to figure this out. 

Jim McGregor: And to be able to find efficiencies that traditionally I didn't find, or that people didn't find,  from [00:31:00] the data that's already there.

Leonard Lee: Yeah, yeah. And,  speaking of edge AI and taking this,  across all these MPUs that are now proliferating across the enterprise edge. It's trying to get, figuring out how to get to good AI, because now,  earlier in the year, we Good AI. Is there good AI? Oh, there's a lot of crap AI out there, trust me, man.

Leonard Lee:  it's getting things out of the experimental stage in terms of the technology, as well as any kind of application architecture that Your generative AI augmented solutions based off of, because like Microsoft earlier this year at build. They were talking about over reliance, right? And this is kind of, uh, alluding to the challenges of quality of a lot of these agents as well as generative AI models, right?

Leonard Lee: Whether you use [00:32:00] RAG or what have you now,  what they introduced at Ignite this year was this term oversharing, which speaks to the security limitations. Of the entire stack. And this is something that I've written about for over a year and talked about for over a year. So as you look at edge AI and taking these large models that have been created through all this massive supercomputing, and you start to bring it down and try to figure out, okay, how do we deploy these things in a good, reliable, safe way, I think that's going to look a lot different than how people are thinking about it right now. 

Leonard Lee: Because we're right now we're thinking about, oh, yeah, you drop a copilot or you run all your stuff on device. Well, depends on how, what kind of data.  you are deploying on device and what you will be dependent on in terms of data [00:33:00] in the rest of the organization and how you manage that,  and so I think all those architectural decisions are going to have implications on where where demand for compute is going to be.

Leonard Lee: And what ultimately what type of chips right and where processing storage will reside and that's where I think edge AI is going to be a really, really funky, funky, weird. Weird phenomenon to observe, uh, in, in the near future because everyone needs to get there. Otherwise, uh, no one's going to be able for all this, especially for, 

Jim McGregor: especially for the, with the use of agents, because, I mean, you have to have, in a lot of cases, you may want traditional AI.

Jim McGregor: Otherwise you have to have some pretty strong guardrails for security, for operational management, for,  all these things for edge AI, where,  you could impact. Your operations, your customers operations, you could impact safety, security, all these different things. [00:34:00]

Leonard Lee: Yeah, 

Jim McGregor: yeah, 

Leonard Lee: yeah. And just like, since we're on a topic of Microsoft, they a couple of things on the silicon side.

Leonard Lee: 1 of the things they introduced was the Azure integrated DPU for storage. Right. So I thought that was interesting because when we think of DPU we, oftentimes we think in terms of processing offload, right, of networking workloads, but here we're seeing something a little bit more akin to, let's say an IPU.

Leonard Lee: Intel's IPU type of concept, but for storage, so 

Jim McGregor: it really depends on who you talk to, because I mean it's, it could be storage. It can be networking. It can be security. It can be even AI workloads. It can be. I mean, it's a, it's an overhead offload engine for the most part. And it's funny because.  we have them from Intel.

Jim McGregor: We have them from NVIDIA. We have them from all the major semiconductor vendors and they're all different. 

Leonard Lee: But  what?  [00:35:00] what's really weird, though?  what's weird? With all this talk about accelerated computing, you have all this offload. But it's to allow the CPU cores to do what they do best.

Leonard Lee: And that doesn't seem to go away. Which is like a funny thing about all of this  sort of accelerated computing offload compute talk.  what I'm saying? And, uh, I don't know, I've been in a couple conversations over the month where people are spelling the doom of the CPU. It's like, well,  the world is more of an XPU kind of universe.

Leonard Lee: And the CPU has its role, . Uh, and so anyway 

Jim McGregor: well, exactly to your point, the CPU is morphing, just like,  I even hesitate to call these GPUs in the data center because let's face it, they're more of an accelerator than, or a graphics processing unit. Yeah, that's fine. Jesse talks 

Karl Freund: about accelerating computing now.

Jim McGregor: Exactly. And the CPUs are going through the same change. I mean, we're seeing different skews especially from Intel and a MD. For different types of [00:36:00] workloads, they're integrating or you look at IBM, they're integrating AI accelerators and some of this, some of this offload functionality into their processors.

Jim McGregor: So I don't think the CPU ever goes away. I think it continues to morph like the rest of the architecture. 

Leonard Lee: Yeah. Yeah. Yeah. But the, thing though is NVIDIA will always, Hey, look at our GPUs, but then what they really mean is. This large, massive, logical heterogeneous blob that, does gen AI supercomputing wonderfully well at a massive scale, right?

Leonard Lee: It, 

Jim McGregor: it now includes their grace cp Yes, ccp. Matter of fact, all three of the companies, Nvidia a MD, and Intel have made the case that the performance of the CPU impacts the AI performance of the system. Over the past couple of months. 

Karl Freund: Yeah. And Jensen's always said he needs top end CPU to make it, to make the GPU sing and dance.

Karl Freund: Right. And so he couldn't get one to talk NVLink after [00:37:00] IBM Power dropped it. Uh, so now he's building his own primarily for NVLink, but also so we can do this fully integrated solution. To, not only today's ai, but for tomorrow's ai. Yeah. They, they should, I always, people, investors ask me, so what is NVIDIA's secret sauce?

Karl Freund: I said, it's their relationship with the researchers that are inventing the next thing.

Mm-Hmm. . 

Karl Freund: Right. That's true. And they, they, they, they're sitting right next to each other. They're, they're working hand in glove, and so they know what's coming next. Much better than the three of us do. 

Jim McGregor: Well, and the meta announcement was a key example of that.

Jim McGregor: Meta has their own chipsets, but they're also working with Nvidia and the fact that they already have their own version of a GB 200 system modified to their specifications. Yeah, 

Leonard Lee: that's right. Yeah. Yeah. Yeah. So,  yeah, I think they should brag about grace a lot more than they do. That is, it is my point,  

Karl Freund: well, Judson does not want.

Karl Freund: Jetson doesn't want the ick [00:38:00] of CPUs. The ick of CPUs is low margin, right?  his margins are well over three times. What a CPU vendor can deliver. He doesn't want to be a CPU vendor. He wants to be a data center vendor. 

Leonard Lee: Yeah. Well, that's, that's probably what he needs to be identify NVIDIA with, because that's literally what they do more than anything else.

Leonard Lee: Revenue share, right. Within their business. So but any,  Last comments or sharing that you gentlemen want to do 

Karl Freund: recipes for Thanksgiving 

Leonard Lee: recipe. Well, ask perplexity. Apparently, right? Is that is that what your family is going to do this year? Carl? 

Karl Freund: Yeah. 

Leonard Lee: Yeah. It's all based on 

Karl Freund: you 

Jim McGregor: know, I would just I would just end with this on my end of the fact that from the enterprise or the user side, we're seeing continued investment in a I,  Anywhere from 15,  the mid teens up into the 30s [00:39:00] and sometimes even 40 percent range in terms of implementing or using AI, whether it's,  their own implementation, whether it's using services and partners.

Jim McGregor: So  I, I think the concept of the AI winter is nowhere near, I think that 2025 be another huge innovative year as we go forward. 

Karl Freund: And I agree. The question that brings to mind is where's all this money going to come from? 

Jim McGregor: That's 

Karl Freund: right. And my theory is, my theory is we're going to see an elongated shelf life of CPUs.

Karl Freund: So the normal refresh rate that we've seen for Intel and AMD CPUs, I'm not sure that's sustainable because money to fund all the AI has got to come from somewhere. That's, that's the fastest way, fastest way to fund all that research and development. 

Leonard Lee: Yeah. I mean, I'll have to slightly beg to differ. I think 2025 is going to be a year where [00:40:00] people discover what this Technology can't do because they imagine so much about what it can do.

Leonard Lee: And so in my own research, mostly focused on and market and how companies above the stack are struggling to really monetize. And help their customers find value in these technologies, largely due to reliability and safe safety issues. I think these POCs are highlighting what this technology, what the limitations of the technologies are.

Leonard Lee: That is something that people haven't discovered yet. And so once you get past that hump, Then you're going to get onto that proverbial Gartner, a plateau of productivity, but before then we're still in a hype cycle because people are, their expectations aren't set on,  have their expectations and perspectives on the technology.

Leonard Lee: Haven't quite yet been checked. By ground truth, right? But 

Jim McGregor: we're in the tech industry. We don't think of [00:41:00] limitations or anything else. We don't think of what could go wrong. We just keep going forward 

Leonard Lee: See, that's what I love about,  it's like two against one,  But I have the power even though I never do it to edit all of your stuff out.

Leonard Lee: So

Leonard Lee: That's why I love about you guys. Hey that's all the time we have. I know that you guys have to jump. You guys have to bounce, but why don't you really quickly tell our audience how they can get in touch with you and tap into the wonderful research resources that both of your firms are. 

Karl Freund: Sure.

Karl Freund: carl. freund at cambrianai. com with a hyphen, as you pointed out. Yes. And, , come see my website, read my writings on Forbes and elsewhere and, have a great holiday. 

Jim McGregor: And gem interiorist research dot com. Obviously look for our partnership with next curb with Forbes with e times, e journal, and a couple of [00:42:00] other publications out there.

Jim McGregor: Both in terms of videos, articles, everything else, and look first at some of the conferences going into 2025 especially if you want to schedule a meeting or a briefing, let us know. 

Leonard Lee: Yes, you must talk to these gentlemen. They're frigging awesome. And that is the official designation of these two gentlemen.

Leonard Lee: Awesome. As well as their firms. And so thank you again, gentlemen. Love doing this with you and yeah, to the audience. Thank you so much for joining us. Please subscribe to our podcast, which will be featured on the next curve YouTube channel. Check out the audio version on buzzsprout. Or find us on your favorite podcast platform.

Leonard Lee: Also subscribe to the next curve research portal@www.next-curve.com for the tech and industry insights that matter. We will see you next time.