적절한 AI 추론 벤치마크 테스트가 필요합니다

We Need A Proper AI Inference Benchmark Test Companies are spending enormous sums of money on AI systems, and we are now at a point where there are credible alternatives to Nvidia GPUs as the compute engines within these systems. Given the amount of money being lavished on these machines and the historically high profits that Nvidia is enjoying – and when I say historical, I mean not for Nvidia, but for the ten or so decades of data processing – competition will just keep coming and eventually there will be several economically viable alternatives. This competition is healthy, and will help drive the innovation that will keep driving the cost of AI processing – particularly AI inference processing – down and down and down until, at some point in the future, it will normalize at a price that doesn’t seem extreme compared to what it has cost in the past decade and what it will cost later this year when new generations of compute engines and networks are brought to bear on large language models. The problem the industry has today is not choice, but trying to figure out what platforms to invest in for their AI training and inference. The demand for AI compute is so much higher than the supply – mostly because of the limited absolute supply of HBM stacked memory and therefore the per-engine capacity – that AI is actually having trouble going mainstream. (It may not feel like this, given all that everyone talks about AI infrastructure these days, but go try to buy a single rack of GPU systems and you will see what I mean.) But as AI mainstreams, and companies other than the hyperscalers and model builders who still drive the bulk of this business figure out how to deploy it as an adjunct to or in place of traditionally programmed applications, they will need to be able to do more rigorous price/performance analysis than is possible today. Which is why I am arguing for someone to be the Jim Gray of the GenAI generation. Or more precisely, I am arguing for the industry to not waste a lot of time arguing and just create a suite of benchmarks that can be used for price/performance analysis across a wide range of architectures and configurations and stop screwing around. We can learn from the past and just skip to the happy ending. The Relational Database Was The GenAI Of Its Time The first big transition from batch to interactive back office systems got its start when Edgar Codd, a researcher at IBM, published a paper called A Relational Model of Data for Large Shared Data Banks, in 1970. This paper outlined how relational databases and how you could interact with them in real time with a structured query language to do crazy correlations that were not easily possible at the time with mainframe and minicomputer systems. IBM’s System R project implemented Codd’s ideas and created the SQL language that is still used today in the mid-1970s, and these ideas were encapsulated in the IBM System/38 minicomputer (which actually used the relational database as its file system) launched in 1978 and eventually were embedded in the DB2 relational database management system for IBM mainframes in 1983. Oracle gets credit for being the first commercial distributor of relational databases, but it is not. But its first commercial version, Oracle V2, did beat the mainframe when it was launched in late 1979 and it opened up the database market when the eponymous database was recoded in C and delivered as Oracle V2 in 1983. And from that moment forward, relational databases went mainstream. When I got into the business in 1989, there were perhaps 40 different computing platforms and maybe 25 different hardware architectures for transaction processing systems. It was an amazing thing to learn, and it has been stunning to see it all consolidate down to Windows Server or Linux on X86 systems, Linux on Arm systems, legacy AIX and IBM i (the great-grandchild of the System/38) on Power, and z/OS on IBM System z mainframes. All of the other proprietary minicomputers and mainframes and all of the other Unix systems are gone. (Technically, deep within Hewlett Packard Enterprise, there is still something that feels like a massively distributed Tandem database cluster running now on X86 iron. But I have no idea if there are any Tandem customers left.) During the Database Wars that raged in the 1980s and 1990s, there were all kinds of metrics of performance, but it was Jim Gray, who was working at Tandem in 1985, which introduced the world to the concept of price/performance in a paper called A Measure of Transaction Processing Power, which also introduced us to a benchmark called DebitCredit that simulated the processing of bank transactions on systems to give us transaction processing throughput as well as cost per transaction to rank systems against each other. There were many criticisms of the DebitCredit benchmark, which simulated the data processing of ten bank branches with a hundred tellers across all of them with 10,000 total accounts to b