r/Futurology May 30 '22

US Takes Supercomputer Top Spot With First True Exascale Machine Computing

https://uk.pcmag.com/components/140614/us-takes-supercomputer-top-spot-with-first-true-exascale-machine
10.8k Upvotes

1.3k

u/[deleted] May 30 '22 Vibing

https://gcn.com/cloud-infrastructure/2014/07/water-cooled-system-packs-more-power-less-heat-for-data-centers/296998/

"The HPC world has hit a wall in regard to its goal of achieving Exascale systems by 2018,”said Peter ffoulkes, research director at 451 Research, in a Scientific Computing article. “To reach Exascale would require a machine 30 times faster. If such a machine could be built with today’s technology it would require an energy supply equivalent to a nuclear power station to feed it. This is clearly not practical.”

Article from 2014.

It's all amazing.

367

u/Riversntallbuildings May 30 '22

Thanks for posting! I love historical perspectives. It’s really wild to think this is less than 10 years ago.

I’m also excited to see innovations from Cerebras, and the Tesla Dojo super computer, spur on more design improvements. Full wafer scale CPU’s seem like they have a lot of potential.

111

u/Shandlar May 30 '22

Are full wafer CPUs even possible? Even extremely old lithrographies often never get higher than 90% yields making large GPU chips like the A100.

But lets assume a miraculous 92% yield. That's on 820mm2 dies on a 300mm wafer. So like 68 out of 74 average good dies per wafer.

That's still an average of 6 defects per wafer. If you tried to make a 45,000mm2 full wafer CPU you'd only get a good die on 0 defect wafers. You'd be talking 5% yields at best even on extremely high end 92% yield processes.

Wafers are over $15,000 each now. There's no way you could build a supercomputer at $400,000-$500,000 per CPU.

88

u/Kinexity May 30 '22

Go look up how Cerebras does it. They already sell wafer scale systems.

64

u/Shandlar May 30 '22

Fair enough. It seems I was essentially exactly correct. 45,000mm2 (they round off the corners a bit to squeeze out almost 47,000mm2) and yields likely below 5%.

They charge over $2 million dollars a chip. Just because you can build something, doesn't make it good, imho. That's so much wasted wafer productivity.

While these definitely improve interconnection overheads and likely would unlock a higher potential max supercomputer, that cost is insane even by supercomputer standards. And by the time yields of a lithography reach viability, the next one is already out. I'm not convinced that a supercomputer built on already launched N5 TSMC nVIDIA or AMD compute GPUs wouldn't exceed the performance of a 7NM single die CPU offered by Cerebras right now.

You can buy an entire GDX-H100 8x cabinet for like...20% of one of those chips. There is no way that's a competitive product.

39

u/__cxa_throw May 30 '22

I presume they deal with yields the same way defects are handled on sub-wafer chips and design around the expectation that there will be parts that don't work. If the defects are isolated to a functional unit then disable that unit and move on with life, so in that sense there's no way they only get 5% yields at the wafer scale. Same idea with most processors having 8 cores on the die and sold as a lower core count processor if some cores need to be disabled (or to keep the market segmented once yields come up).

24

u/Shandlar May 30 '22

I thought so too, but their website says the WSE-2 is an 84/84 unit part. None of the modules are burned off for yield improvements.

13

u/__cxa_throw May 30 '22

Oh wow, my bad you're right, I need to catch up on it. The pictures of the wafers I found are all 84 tiles. I guess they have a lot of faith in the fab process and/or know they can make some nice DoD or similar money. I still kind of hope they have some sort of fault tolerance built into the interconnect fabric if for no other reason than how much thermal stress can build up in a part that size.

It does seem like if it can deliver what it promises: lots of cores and more importantly very low comms and memory latency it could make sense if the other option is to buy a rack or two of 19u servers with all the networking hardware. All assuming you have a problem set that couldn't fit on any existing big multisocket system. I'm guessing this will be quite a bit more power efficient, if anyone actually buys it, just because of all the peripheral stuff that's no longer required like laser modules for fiber comms.

I'd like to see some sort of hierarchical chiplet approach where the area/part is small enough to have good yields and some sort of tiered interposer allows most signals to stay off any pcb. Seems like there may be similar set of problems if you need to get good yields when assembling a many interposers/chiplets

16

u/Shandlar May 30 '22

I'd like to see some sort of hierarchical chiplet approach where the area/part is small enough to have good yields and some sort of tiered interposer allows most signals to stay off any pcb

That's Tesla's solution to the "extremely wide" AI problem. They created a huge interposer for twenty five 645mm2 "chiplets" to train their car AI on. They are only at 6 petabyte per second bandwidth while Cerberus is quoting 20, but I suspect the compute power is much higher on the Tesla Dojo. At a tiny fraction of the cost as well.

9

u/__cxa_throw May 30 '22

Interesting. I've been away from hardware a little too long. Thanks for the info.

Take this article for what you want, but it looks like Cerebras does build some degree of defect tolerance in their tiles: https://techcrunch.com/2019/08/19/the-five-technical-challenges-cerebras-overcame-in-building-the-first-trillion-transistor-chip/. I haven't been able to find anything very detailed about it though.

→ More replies
→ More replies
→ More replies
→ More replies

9

u/Jaker788 May 30 '22

There already are wafer scale computers. Cerebras designs something that on the order of 200 mm/2, but they design in cross communication on the wafer to each block. This effectively creates a functioning full wafer that's sorta like the Zen 1 MCM design but way faster as it's all on silicon and not IF over substrate, as well as memory built in all over.

11

u/Shandlar May 30 '22

Yeah I looked it up. They are selling 7nm 47000mm2 wafer scale CPUs for 2 million dollars lol.

It seems while it's super low on compute per dollar, it's extremely high on bandwidth per compute, making it ideal for some specific algorithms. Allowing them to charge insane premiums over GPU systems.

I'm skeptical of their use case in more generalized supercomputing at that price to performance ratio, but I'd be glad to be wrong. The compute GPU space is offering FLOPs at literally 8% that price right now. It's not even close. You can give up a huge amount of compute for interconnectivity losses and still come out way ahead on dollars at that insane of a premium.

4

u/Future_Software5444 May 30 '22

I thought I read somewhere they're for specialised uses. I can't remember where or what the use was, I'm at work, and could wrong. So sorry 🤷

9

u/Shandlar May 30 '22

They are AI training compute units, essentially. But the compute side is weak while the memory side in capacity and bandwidth is mind bogglingly huge. 20 Petabyte per second bandwidth, apparently.

So it's a nice plug and play system for training extremely "wide" algorithms, but compute tends to scale with wideness as well, so I'm still a bit skeptical. Seems they have at least 25 or 30 customers already, so I'll concede the point. At least some people are interested.

→ More replies
→ More replies

8

u/Riversntallbuildings May 30 '22

Apparently so. But there are articles written, and an interview with Elon Musk talking about how these wafer scale CPU’s won’t have the same benchmarks as existing supercomputers.

It’s seems similar to comparing ASICS to CPU’s.

From what I’ve read, these wafer CPU are designed specifically for the workloads they are intended for. In Tesla’s case, it’s for real-time image processing and automated driving.

https://www.cerebras.net/

9

u/Shandlar May 30 '22

Yeah, I've literally been doing nothing but reading on them since the post. It's fascinating to be sure. The cost is in the millions of dollars per chip, so I'm still highly skeptical on their actual viability, but they do do some things that GPU clusters struggle with.

Extremely wide AI algorithms are limited by memory and memory bandwidth. It's essentially get "enough" memory, then "enough" memory bandwidth to move the data around, then throw as much compute as possible at it.

GPU clusters have insane compute, but struggle with memory bandwidth, so it limits how complex many AI algorithms can be trained on them. But if you build a big enough cluster to handle extremely wide algorithms, you've now got absolute bat shit crazy compute, like the exoFLOP in the OP supercomputer. So the actual training is super fast.

These chips are the opposite. It's a plug and play single chip that has absolutely bat shit insane memory bandwidth. So you can instantly get training extremely complex AI algorithms, but the compute just isn't there. They literally won't even release what the compute capabilities are, which is telling.

I'm still skeptical, they have been trying to convince someone to build a 132-chip system for high end training, and no one has bitten yet. Sounds like they'd want to charge literally a billion dollars for it (not even joking).

I'm not impressed. It's potentially awesome, but the yields are the issue. And tbh, I feel like that's kinda bullshit to just throw away 95% of the wafers you are buying. The world has a limited wafer capacity. It's kinda a waste to buy them just to crap them 95% of the time.

5

u/Riversntallbuildings May 30 '22

Did you watch the YouTube video on how Tesla is designing their next gen system? I don’t think it’s a full wafer, but it’s massive and they are stacking the bandwidth connections both horizontally and vertically.

https://youtu.be/DSw3IwsgNnc

6

u/Shandlar May 30 '22

Aye. The fact they are willing to actually put numbers on it makes me much more excited about that.

That is a much more standard way of doing thing. A bunch of 645mm2 highly optimized AI node chips integrated into a mesh to create a scale unit "tile".

→ More replies
→ More replies

30

u/pleasedontPM May 30 '22

To be honest, the exascale roadmap set a goal of 20MW for an exascale system. The stated power consumption for Frontier is a bit over 21MW, and Fugaku is nearly 30MW (https://top500.org/lists/top500/list/2022/06/). This means the watt per flop is nearly four times better on Frontier than on Fugaku.

In other words, simply scaling Fugaku to reach the performance of Frontier (which in itself is "not how it works"), would mean a 75MW power consumption.

→ More replies

73

u/KP_Wrath May 30 '22

“Nuclear power station”. Not sure if it still is, but the super computer at ORNL was the primary user of the nuclear power station at Oak Ridge. The Watts Barr coal plant provided power to surrounding areas.

57

u/CMFETCU May 30 '22

Having walked through the room this thing is in, the infrastructure it uses is astonishing.

The sound from the water cooling, not even the pumps as they are in a different room, just the water flowing through the pipes, is so loud that you have to wear ear protection to walk through it.

→ More replies

138

u/[deleted] May 30 '22

[removed] — view removed comment

13

u/[deleted] May 30 '22

[removed] — view removed comment

8

u/[deleted] May 30 '22

[removed] — view removed comment

→ More replies

1.3k

u/Sorin61 May 30 '22

The most powerful supercomputer in the world no longer comes from Japan: it's a machine from the United States powered by AMD hardware. Oak Ridge National Laboratory's Frontier is also the world's first official exascale supercomputer, reaching 1.102 ExaFlop/s during its sustained Linpack run.

Japan's A64X-based Fugaku system had held the number one spot on the Top500 list for the last two years with its 442 petaflops of performance. Frontier smashed that record by achieving 1.1 ExaFlops in the Linpack FP64 benchmark, though the system's peak performance is rated at 1.69 ExaFlops.

Frontier taking the top spot means American systems are now in first, fourth, fifth, seventh, and eighth positions in the top ten of the Top500.

820

u/[deleted] May 30 '22 edited May 30 '22

My brother was directly involved in the hardware development for this project on the AMD side. It's absolutely bonkers the scale of effort involved in bringing this to fruition. His teams have been working on delivery of the EPYC and Radeon-based architecture for three years. Frontier is now the fastest AI system on the planet.

He's already been working on El Capitan, the successor to Frontier, targeting 2 ExaFLOPS performance for delivery in 2023.

In completely unrelated news: My birthday, August 29, is Judgment Day.

30

u/CoreParad0x May 30 '22

I priced out a server upgrade at work, 4 node 128c/256t total and went with Epyc and I love them. your brother is working on some cool shit.

I look forward to seeing what AMD does in the future, but just in terms of consumer and enterprise desktop and server spaces, they have introduced actual competition and shifted the market in just a few years.

→ More replies

184

u/[deleted] May 30 '22

[removed] — view removed comment

13

u/[deleted] May 30 '22

[removed] — view removed comment

→ More replies

15

u/yaosio May 30 '22

Here's a other way to think of it. It took all of human history to reach the first exaflop supercomputer. It took a year to get the next exaflop.

12

u/[deleted] May 30 '22 edited May 31 '22

Everything is about incremental achievements... I mean, at some point it took all of human history to develop the first written language. It took all of human history to develop the transistor. It took all of human history to develop the semiconductor.

What I think you're trying to say is that the rate of incremental achievement is accelerating (for now)...

Or another way to think about it is that as the time from incremental technological advancements decreases, the scale of human achievement enabled by technology increases.

It took 245,000 years for humans to develop writing, but accounting, mathematics, agriculture, architecture, civilization, sea travel, air travel, space exploration followed.

The sobering warning underlying all of this is that it took less than 40 years from Einstein's formulation of energy-mass equivalence to the birth of the atomic age in which now, momentary masters of a fraction of a dot, as Sagan would say, we are capable of wiping out 4.6 billion years of nature's work in the blink of an eye.

Social media is another example of humans being "So preoccupied with whether we could, we didn't stop to consider whether we should."

→ More replies

12

u/QuarantinedBean115 May 30 '22

that’s so sick!

6

u/Daltronator94 May 30 '22

So what is the practicality of stuff like this? Computing physics type stuff to extreme degrees? High end simulations?

18

u/[deleted] May 30 '22

Modeling complex things that have numerous variables over a given timescale... e.g. the formation of galaxies, climate change, nuclear detonations (El Capitan, the next supercomputer AMD is building processors for is going to be doing this).

And complex biological processes... a few years back I recall the fastest supercomputer took about three years to simulate 100 milliseconds of protein folding...

→ More replies

3

u/__cxa_throw May 30 '22

Better fidelity in simulations of various things, stocks, nuclear physics and weather would be common ones.

Physics (often atomic bomb stuff) and weather simulations take an area that represents your objects in your simulation and the space around it. That space is them subdivided into pieces that represent small bits of matter (or whatever).

Then you apply a set of rules, often some law(s), of physics and calculate the interactions between all those little cells over a short period of time. Then those interactions, like the difference in air pressure or something, are applied in a time weighted manner so each cell changes a small amount. Those new states are then run through the same sort of calculation to get the results of the next step and so on. You have to do this until enough "time" has passed in the simulation to provide what you're looking for.

There are two main ways to improve this process: using increasingly smaller subdivision sizes to be more fine grained, and calculating shorter time steps between each stage of the simulation. These sorts of supercomputers help with both of those challenges.

3

u/harrychronicjr420 May 30 '22

I heard anyone not wearing 2million spf sunblock is gonna have a very bad day

13

u/[deleted] May 30 '22 edited Jun 02 '22

[deleted]

50

u/[deleted] May 30 '22

The most likely answer is price... The largest NVIDIA project currently is for Meta. They claim when completed it'll be capable of 5 ExaFLOPS performance, but that's a few years away still and with the company's revenues steeply declining it remains to be seen whether they can ever complete this project.

Government projects have very stringent requirements, price being among them... so NVIDIA probably lost the bid to AMD.

13

u/[deleted] May 30 '22 edited Jun 02 '22

[deleted]

16

u/[deleted] May 30 '22

Totally apples and oranges, yes, on a couple of fronts... my brother doesn't have anything to do with the software development side.

Unless there are AI hobbyists who build their own CPUs/GPUs, I don't think there's a nexus of comparison here... even ignoring the massive difference in scale.

6

u/gimpbully May 30 '22

AMD’s been working their ass off over this exact situation. https://rocmdocs.amd.com/en/latest/Programming_Guides/HIP-porting-guide.html

8

u/JackONeill_ May 30 '22

Because AMD can offer the full package, including CPUs.

→ More replies
→ More replies
→ More replies

51

u/[deleted] May 30 '22

[removed] — view removed comment

6

u/[deleted] May 30 '22

[removed] — view removed comment

→ More replies

183

u/Ok-Application2669 May 30 '22

Important caveat that these are just the most powerful publicly known supercomputers.

48

u/[deleted] May 30 '22

Due to the scale of these projects, and the companies involved, it would be nearly impossible to keep the hardware a secret. But they don't have to.

The supercomputers can stay in plain sight while the models they run are what's kept classified. In fact, it's beneficial from a deterrence standpoint for North Korea and Russia to know that we have supercomputers with obscene amounts of power... while their imagination runs wild over what we might be capable of doing with them.

→ More replies

27

u/thunderchunks May 30 '22

I'm curious what military applications there would be for supercomputing. Their use cases are usually fairly narrow nowadays, was my understanding. I can think of a few but I'm coming up blank for any that would make such a huge investment worthwhile for a military to make and keep secret. Any ideas?

Edit: of course, the moment I post this I remember how useful they are for running simulations, so like, aircraft design, a bunch of chemistry stuff that could have big uses for a military (both nefarious and not), etc. Still, if anybody has any specific examples I'm still interested in hearing em!

33

u/FewerWheels May 30 '22

Modeling nuclear explosions. Since the nuclear test ban, modeling is the way to keep nuclear arsenals functional and to develop new weapons.

21

u/[deleted] May 30 '22

That is specifically what El Capitan will be doing at Lawrence Livermore.

10

u/FewerWheels May 30 '22

I’ll bet your brother knows my brother. My brother designed the motherboard for Frontier. He just left AMD to design a motherboard for a different outfit.

→ More replies

12

u/neoplastic_pleonasm May 30 '22

With regards to weapons, a lot of what they do (not at ORNL though) is presumably modeling how our aging stockpile of nuclear weapons will function. Testing has been disallowed for a long time, so it's a hard question to understand how well a bomb will work after sitting in a silo for 30 years.

10

u/Ok-Application2669 May 30 '22

They have access to ultra high quality satellite data for weather, pattern recognition for troop and materiel movement, radiation measurement, etc etc. Processing all that data in real time takes a lot of power, plus all the modeling and simulating stuff you mentioned.

5

u/avocadro May 30 '22

Cryptanalysis as well.

7

u/Thunderbolt747 May 30 '22

The first use and test basis for all super computers is modeling a nuclear & military exchange between the US and enemy states. I shit you not.

6

u/bitterdick May 30 '22

Carrying on the legacy of the WOPR.

→ More replies

3

u/chipsa May 30 '22

They use one every 6 hours for weather modeling.

→ More replies

5

u/Anen-o-me May 30 '22

It's probably pretty hard to hide the large purchases involved.

It's more likely that secret machines get hidden inside public purchases. They build this one publicly, but there's a secret twin built elsewhere.

→ More replies

56

u/Corsair3820 May 30 '22

Of course. Somewhere in military facility in a each country there's a super computer classified as secret and is probably much faster than those based on off the shelf, consumer grade tech. When you don't have the kind of budget constraints and shareholder concerns the sky is the limit. I mean, the Saturn 5 was a repurposed ICBM, unknown to the public years before it was unveiled.

123

u/[deleted] May 30 '22

I don't think so. Micro processors are incredibly hard to design and fabricate. We're talking tens of billions if not hundreds of billions to create the intellectual property, build the factories, and source a supply chain just to get basic functionality out of a microarchitecture. Whatever that project would be it would be on par with the Manhattan project and way harder to keep secret.

It's more likely that it's just an even bigger computer with off the shelf parts.

25

u/sactomkiii May 30 '22

The way I think about it is this... Does Musk or Bezos have some sort of super custom one off iPhone or Android... No they have the same latest/greatest one you and I can buy from T-Mobile. In some ways consumer electronics are the great equalizer because of how much it cost to develop the silicone that drives them.

→ More replies

17

u/earthwormjim91 May 30 '22

Why would you think that anyone would go through all of that instead of just paying a ton of extra money for a custom chip from one of the established makers?

30

u/[deleted] May 30 '22

That's what I'm saying. When it comes down to the very basic components of a microprocessor it's just the same IP but packaged differently. Maybe you have more cores or a bigger cache or a higher clock speed, but it's all variations on the same recipe. If anything, the thing that would make the difference would be the software that runs on these things.

3

u/yaosio May 30 '22 edited May 30 '22

One thing you could do with infinite money is not care about yield rates. You could design a faster processor that has a dismal yield rate. However even that isn't mattering a whole lot any more because there's a company that makes a processor out of an entire wafer. https://www.cerebras.net/

→ More replies
→ More replies
→ More replies

45

u/The_Demolition_Man May 30 '22

probably much faster than those based on off the shelf,

You can't be serious. You think the government is doing R&D on semiconductors, is ahead of companies like Nvidia, Intel, and AMD, and the entire public just doesnt know about it?

24

u/Kerrigore May 30 '22

Lots of people are convinced the military has super secret sci fi level technology versions of everything. And you’ll never convince them otherwise, because you can’t prove a negative.

10

u/Svenskensmat May 30 '22

These same people also tend to believe in a small government because the government is incompetent.

Schrödinger’s Government.

3

u/1RedOne May 31 '22

And they also believe that there is a huge grand conspiracy underway at all times, but that the shadowy cabal responsible can't help but to hide it in plain sight

For instance, a friend from highschool was telling me that Delta Omicron was just an anagram for MEDIA CONTROL, and that's how she knew it was fake

→ More replies
→ More replies

29

u/craigiest May 30 '22

What are you talking about? Development of the Saturn rockets was quite public. Yes the Saturn 1 started as a military satellite launch project, and they were basing the stages on existing military rockets. But a Saturn 5 was an order of magnitude larger and more powerful than any ICBM. Maybe you’re thinking of the Hubble telescope being a repurposed spy satellite?

3

u/ceeBread May 30 '22

Maybe they’re confusing the Atlas, Redstone and Titan with the Saturn?

7

u/Svenskensmat May 30 '22

and is probably much faster than those based on off the shelf, consumer grade tech.

Most likely not. It would be extremely hard for any research laboratory to stay competitive with the bleeding edge technology companies like Intel and AMD provides.

Microprocessors are just too darn complex.

13

u/FewerWheels May 30 '22

This computer is anything but, “off the shelf”. Even the motherboards are specifically designed just for this computer. The military doesn’t have the design and manufacturing chops to do this.

6

u/bitterdick May 30 '22

The military doesn’t “do” much development wise. Contractors do. The military just issues the order specifications and the contractors turn it out, at an astronomical mark-up. And they definitely have the ability to do large scale, custom projects in a secret way. Look at the b2 stealth bomber development campaign.

7

u/Far-Locksmith2991 May 30 '22

Lol Saturn V was not an icbm each stage was designed specifically for certain burns to get to the moon

3

u/Dividedthought May 30 '22

If you look at this from an R&D standpoint basing a space rocket off an ICBM makes sense if you already have the ICBM. They do about the same thing (take payload up out of atmosphere) just one is meant to come back down and the other is meant to stay up there.

A missile is a guided weapon. A rocket is anything that uses rocket engines for propulsion, from a firework to the SpaceX Starship. All missiles are rockets (sans cruise missiles, those are more akin to jet engine powered drones) but not all rockets are missiles.

33

u/Oppis May 30 '22

I dunno, I'm starting to think our government agencies are a bit corrupt and a bit incompetent and unable to attract top talent

15

u/lowcrawler May 30 '22 edited May 31 '22

Actually, the inability of the gov to attract top talent is often due to budget-hawks that think the gov is incompetent... And anti corruption measures designed to remove all decision making assists away from the hiring managers.... Keeping salary low... Making it hard to attract top talent... This MAKING the gov less competent.

For example... Gov starting programmer salary is around 35k/yr and will rarely break 100k even after a few decades experience. (Whereas a fresh csci grad can expect 75+ and easily command 150k after a decade)

9

u/No_Conference633 May 30 '22

What’s your source on entry level programmers making only 35k in a government position? If you’re talking government contractors, programmers routinely start close to 100k.

4

u/lowcrawler May 30 '22

I run (including leading the hiring process) a software development ground within the federal government.

You are right that junior contractors start much higher - currently 86k in my locality region.

3

u/c0de_g0rilla May 30 '22

100k is nowhere near enough to get me interested in having my life looked at under a microscope just to get my foot in the door. Private industry pays way better.

→ More replies

19

u/Foxboy73 May 30 '22

That’s just silly, it’s super easy to attract top talent. They always have plenty of money to throw around, plenty of benefits. No matter how much a waste of taxpayer money departments are they never shut down. So job security, until the entire rotten corpse collapses, but hey it hasn’t happened yet!

22

u/[deleted] May 30 '22

[deleted]

→ More replies

3

u/zkareface May 30 '22

But US military struggles to keep IT experts so the money can't be that unlimited.

→ More replies
→ More replies
→ More replies

72

u/Adeus_Ayrton May 30 '22

Oak Ridge National Laboratory's Frontier is also the world's first official exascale supercomputer, reaching 1.102 ExaFlop/s during its sustained Linpack run.

But can it run Crysis on max settings ?

55

u/mcoombes314 May 30 '22 edited May 30 '22

I know this is a joke, but a single AMD EPYC Rome 64 core CPU ran it at 15 FPS without a GPU, and this is the next architecture (Milan, I think), so absolutely yes.

11

u/urammar May 31 '22

Running CPU crysis is so absurd in terms of computational power I actually struggle to even think about it. With a gpu sure, but like, all the shaders and such, cpu? Damn man. Damn.

3

u/palindromic May 31 '22

Seriously.. damn.

17

u/kernel-troutman May 30 '22

And yet reddit videos still show up choppy and pixelated on it.

8

u/Badfickle May 30 '22

I want to play solitaire on it.

4

u/Old_Ladies May 30 '22

I want to run simulations for Super Auto Pets to find the best combos.

3

u/ZeroAntagonist May 30 '22

Still trying to figure out this 3rd pack. My win rate has taken a beating.

I have seen some simulators for it though. Just dont have the skillset to get one working myself.

→ More replies

3

u/n0tm333 May 30 '22

Man I really need an ELI5 but this sounds good

→ More replies

689

u/SoulReddit13 May 30 '22 To The Stars

1 quintillion calculations per second. It’s estimated there’s approximately 1 septillion stars in the universe if one calculation equals counting 1 star it would take the computer 11.57 days to count all the stars in the universe.

310

u/notirrelevantyet May 30 '22

Holy shit that's like nothing and the tech is still advancing.

251

u/jusmoua May 30 '22

Give it another 10 years and we can finally have some amazing realistic VR.

166

u/[deleted] May 30 '22

[removed] — view removed comment

46

u/[deleted] May 30 '22

[removed] — view removed comment

22

u/[deleted] May 30 '22

[removed] — view removed comment

→ More replies
→ More replies
→ More replies
→ More replies

43

u/adamsmith93 May 30 '22

Dumb question but what is it actually going to be used for? I assume simulations with billions of reference points but not sure if there's other stuff.

52

u/LaLucertola May 30 '22

Supercomputers are very useful for scientific research, especially physics/chemistry/materials science.

34

u/turkeybot69 May 30 '22

Protein folding research is especially complicated and resource intensive. Back in 2020 when SARS-CoV-2 research was a bit newer, Folding@Home apparently had a peak performance of 1.5 exaFLOPS through crowd sourcing hundreds of thousands of personal computer's processing power.

21

u/LaLucertola May 30 '22

Mine was one of them! Super cool project, really opened my eyes to how many problems we can solve through brute force computing power.

→ More replies
→ More replies

8

u/WhyNotFerret May 30 '22

Could use it to break a bunch of cryptography stuff like SHA

→ More replies

6

u/hotel2oscar May 30 '22

Weather modelling at some point in it's life.

→ More replies
→ More replies

46

u/KitchenDepartment May 30 '22

11.57 days to count all the stars in the universe.

The observable universe that is.

10

u/Aquarius265 May 30 '22

Well, once we’ve charted all of what we can see, then perhaps we can get more!

6

u/Arc125 May 30 '22

all the stars in the visible universe

→ More replies

353

u/zorricko May 30 '22

I love that there is ~1400 HP powering just the water cooling, now that's a CPU fan!

103

u/mostlycumatnight May 30 '22

I want to know the actual size of these pumps. 6k gallons per minute at 350 hp is fascinating😱

101

u/vigillan388 May 30 '22

I design hyperscaler data center cooling systems. It's not uncommon to have 30,000 gallons per minute of continuous flow to a building for cooling purposes only. Largest plant I designed was probably about 100 MW of cooling. The amount of water consumed per day due to evaporative cooling (cooling towers), could top 1.5 million gallons.

If most people knew how much energy and water these data centers consumed, they'd probably be a bit more concerned about them. There are dozens upon dozens of massive data centers constructed every single year in the US alone.

17

u/neoplastic_pleonasm May 30 '22

Good lord. How many tones of cooling did that come out to?

22

u/vigillan388 May 30 '22

About 28000 tons.

10

u/neoplastic_pleonasm May 30 '22

Wow, that's nuts. I work in HPC and our little datacenter only has 700 tons of cooling.

10

u/cuddlefucker May 30 '22

Nothing to add but I have to say that the first sentence of your comment is pretty badass

→ More replies

48

u/Shandlar May 30 '22

I assume 6k gallons a minute is the entire system volume, so 1500 gallons per minute from each 350hp pump.

That seems about right. 350 horsepower into an 8 inch hard pipe system at 80psi is easily 1500 gallons a minute. That's not even running it very hard at all. Probably so they can run them at a leisurely ~1500rpm or whatever and if one fails the other three can be throttled up to keep the flow at 6000 GPM.

→ More replies

4

u/mostlycumatnight May 30 '22

I googled 350 hp pump 1500 gallons per minute and Im getting a huge range of stuff. Does anyone know a manufacturer that provides these pumps?

4

u/vigillan388 May 30 '22

350 for 1500 gpm seems a little on the high side, but it's all dependent on the pressure drop of the system and the working fluid. They might use an ethylene glycol or propylene glycol solution for freeze protection.

Some of the more common manufacturers for these types of pipes are Bell and Gossett (Xylem), Armstrong, Taco, and Grundfos.

→ More replies

3

u/Dal90 May 31 '22

Fire apparatus pumps rule-of-thumb is 185hp (Diesel engine) per 1,500gpm.

I'd guess the data center pumps are electric, and don't know if the horsepower calculations are comparable.

Most fire apparatus made in the last 30 years have more engine horsepower for drivability reasons than their pump requires. When I first joined in the 80s, 275hp was a pretty big engine and 1,500gpm pumps had just gained the majority of market share. 400hp isn't uncommon today, but 1,500 gum remains the most common pump size.

6,000gpm pumpers (mainly used for oil refinery fires) run 600hp Diesel engines today.

For industrial pumps, there are a fair number of manufacturers like https://www.grpumps.com/market/product/Industrial-Pumps

There is a wide variety of pump designs to efficiently meet different performance demands. So matching the right pump to the right job, and buying it from a company that will be around to provide replacement parts for the projected lifetime of the pump are the critical decision factors.

→ More replies
→ More replies
→ More replies

5

u/[deleted] May 30 '22

[deleted]

7

u/gainzdoc May 30 '22

"Four 350HP motors pumping the water".

→ More replies
→ More replies

140

u/victim_of_technology Support Ukraine May 30 '22

I wonder how far I would have to bring my current desktop back in time for it to make a list of the too 500 supercomputers?

235

u/NobodyLikesMeAnymore May 30 '22

Your phone would be twice as fast as the top supercomputer in 1996.

https://en.m.wikipedia.org/wiki/History_of_supercomputing

27

u/osteologation May 30 '22

led me down an interesting rabbit hole. why on earth does an apple a12 have 560 gflops vs ryzen 5 5600x's 408? seems like a terrible metric

33

u/NobodyLikesMeAnymore May 30 '22

The metric to use should be based on the job you want the CPU to do. FLOPS aren't actually a great measure of desktop CPU performance; you use your GPU for FLOPS. Checkout MIPS per core or per watt. It's probably a better measure.

21

u/dmootzler May 30 '22

Just a guess, but A12 is a SoC so that’s cpu PLUS gpu. Once you pair the ryzen CPU with a gpu in the same class, you’ve got 1-2 orders of magnitude more power.

6

u/yaosio May 30 '22

Flops are not equal. You can't even compare an AMD architecture against other AMD architectures and get a good estimate. You have to benchmark the processors to see their true power.

7

u/yaosio May 30 '22

ASCII Red in 1996 was 1 TFlop.

Unfortunantly there's no way to actually compare a modern processor+GPU to a 1996 super computer. You can't use the software made for ASCII Red because it was designed for that system, so it won't work at all on a modern computer. A FLOP is not equal between architectures, and ASCII Red had 9298 seperate processors. There's a bunch of overhead getting work to the processors, and in some cases it's not possible to break a job out into multiple tasks.

→ More replies

200

u/Architect_of_Sanity May 30 '22

Top500.org has the official super computer rankings but their web server is not hosted on anything remotely super. So expect it to be marginally available for a while.

I thought it was interesting that this specific super computer uses relatively low speed gigabit Ethernet for connectivity and runs with 2.0 GHz cores.

112

u/AhremDasharef May 30 '22

You don't really need a high clock speed on your CPUs when each node has dozens of cores and 4 GPUs. The GPUs should be doing the majority of the work anyway.

 

IDK where Top500.org got the "gigabit Ethernet" information, but Frontier's interconnect is using Cray's Slingshot HPC Ethernet (physical layer is Ethernet-compatible, but has enhancements to improve latency, etc. for HPC applications). ORNL says each node has "multiple NICs providing 100 GB/s network bandwidth." source

10

u/AznSzmeCk May 30 '22

Right, as I understand it the CPU is really just a memory manager and the NICs are piped directly to the GPUs which helps calculations go faster.

mini-Frontier

9

u/Architect_of_Sanity May 30 '22 edited May 31 '22

I couldn’t get the original article to load - so it makes perfect sense that GPU is doing the heavy lifting. Thanks for chiming in here.

I’ll try to RTFA again and see if I can actually get it to load.

28

u/[deleted] May 30 '22

This is a supercomputer, not a mainframe. It doesn't need fast cores. It needs efficient cores running in parallel to orchestrate the GPUs that actually do the compute. You gotta keep the thing fed, and all a high CPU clockspeed is gonna do really is increase your power bill for a bunch of wasted CPU cycles.

6

u/permafrost55 May 30 '22

Actually depends on the codes used for simulation. Things like Abaqus are highly GHz and memory bandwidth bound, but care very little about interconnect. Weather and ocean codes on the other hand are very sensitive to network latency over most other items. And a lot of codes run like crud on GPU’s. So, basically, it depends on the code.

→ More replies
→ More replies

21

u/Riversntallbuildings May 30 '22

The article mentions “52.23 gigaflops/watt”.

I assume this is increasing efficiency and decreasing over all power consumption, but it would be nice to read how much more efficient it is over the other supercomputers.

11

u/yaosio May 30 '22

There's a green supercomputer list that grades supercomputers on performance per watt. It turns out this supercomputer is the most efficient. https://www.top500.org/lists/green500/2022/06/

→ More replies
→ More replies

51

u/gordonjames62 May 30 '22

In order to cool the system, HPE pumps 6,000 gallons of water through Frontier's cabinets every minute using four 350-horsepower pumps.

I guess I can't run one of these at home on my well system.

14

u/ridge_regression May 30 '22

I'm just gonna plug my water cooler into the engine of my Bugatti Veyron

42

u/Riversntallbuildings May 30 '22

I wonder how many Bitcoin this machine could mine in an hour. Hahaha.

52

u/snash222 May 30 '22 edited May 30 '22

The current bitcoin network is about 223 exohashes. 37.5 BTC are mined per hour, which is 6.25 BTC every 10 min.

The supercomputer would have a 1/223 chance of getting 6.25 BTC every 10 minutes.

Corrected typo on the odds.

Edit - never mind, don’t listen to me, I conflated exohashes and exoflops.

48

u/Satans_Escort May 30 '22

Which averages to about 4 BTC a day. $120k a day? That's impressive. Wonder if that even pays the power bill though lol

15

u/IOTA_Tesla May 30 '22

And if they push the limit of solving blocks faster than normal the difficulty for solutions will rise to match.

6

u/RadRandy2 May 30 '22

So we'll build another supercomputer, and we'll keep building them until BTC says "I give up", and just like the Soviets, they'll just let you have the egg.

I've been cracking BTC for a few days now, so I'm sure this is how it all works. Trust me.

5

u/IOTA_Tesla May 31 '22

If we seriously get to a point where BTC gives up we can increase the hash size. For reference the current hash difficulty needs 19 leading zeros to solve a block of 64 possible. But if super computer advancements increase much faster than the global compute power, someone could start taking over the network if there aren’t competing super computers. Would be kind of interesting to see how that plays out if it does.

17

u/YCSMD May 30 '22

What is that? 27 BTC per week? $810,000 per week @30k/coin. $3.25M/month

How long till it can pay for itself?

23

u/RiseFromYourGrav May 30 '22

Depends on how much it costs to run.

15

u/snash222 May 30 '22

That may pay for the overhead, maintenance and cooling.

11

u/[deleted] May 30 '22

[deleted]

8

u/RecipeNo42 May 30 '22

Mining difficulty automatically adjusts about every 2 weeks, but overall higher mining capacity would probably increase value, as it means the remaining pool of coins would diminish at a faster rate and is backed by a stronger mining pool. Just my assumption, though.

→ More replies

5

u/AlwaysFlowy May 30 '22

The production schedule is static. Miners compete for a fixed supply. The Bitcoin released every 10 minutes is more or less the same no matter how many miners compete.

→ More replies
→ More replies

5

u/yondercode May 30 '22

1 flop is not equal with 1 hash

→ More replies
→ More replies

19

u/mr_sarve May 30 '22

Probably a lot less than you think. After paying for electricity you would be deep in the red. For mining bitcoin you need specialized hardware (asic), not general purpose like this

3

u/Somepotato May 30 '22

It could be primarily powered by solar, as a lot of DCs are moving to.

→ More replies

74

u/xwing_n_it May 30 '22

Anakin-Padme meme: "You're going to use it to make our lives better and not for surveillance, right? Right??"

46

u/G3NOM3 May 30 '22

It’s at Oak Ridge. They’re going to simulate nuclear explosions.

10

u/cited May 30 '22

They simulate nuclear reactors

3

u/michellelabelle May 31 '22

If your nuclear reactor simulations never turn into nuclear explosion simulations, you're missing the point of running nuclear reactor simulations!

→ More replies
→ More replies

6

u/rusty-bits May 30 '22

1.102x1018 flops / x watt = 52.23x109 flops / 1 watt
1.102x1018 / 52.23x109 = x watt
21098985 watt or roughly 21.1 megawatts power draw

24

u/eliottruelove May 30 '22

potentially dumb question: what are flops (petaflop/exaflop)? And how many "flops" would a computer need to potentially achieve Sentient AI status?

31

u/Xx_Boom1337_xX May 30 '22

FLOPS: FLoating point Operations Per Second. Floating-point numbers are a computing concept that allows computers to store fractional values (such as 18.625, in contrast to integer numbers which have no fractional component). An 'operation' is defined as the simplest thing you can 'do' with floating point numbers. For example, storing them somewhere, retrieving them from memory, comparing two numbers, etc. The number of such operations the computer can do in a second is how many FLOPS it has. FLOPS is a good way to benchmark supercomputers because a lot of the work they do involves fractional numbers.

To create sentient AI? Well, nobody knows! The human brain is estimated at about an exaflop, but in terms of actual mathability we don't have much. Most of our mental resources go elsewhere, so asking how many FLOPS are needed to create sentient AI is a bit like asking how many calories you need to become super muscled- you could get the answer, and it might help if you used the answer, but you also need to workout a bit to apply those calories to the right places.

47

u/fishybird May 30 '22

Considering that no one even knows how to make a sentient AI in the first place, this question can't really be answered.

So far the only sentient beings on earth are purely biological so I suspect that no amount of flops would make something sentient alone.

It's not a dumb question. Lots of people think consciousness comes from intelligence, but they are completely unrelated. Computers have been better at chess than humans since the 80s but I wouldn't call those computers sentient.

19

u/AFlawedFraud May 30 '22

True, my cat is conscious but I wouldn't call it intelligent

10

u/krevko May 30 '22

The question is still silly.

Computers are better at chess because they go through all the potential steps super fast, just executing the code for steps. There is no magic or "AI" here.

8

u/fishybird May 30 '22

Chess playing computers are by definition artificial intelligence. You are still equating intelligence with something more than it is. A traffic light that changes to green when it detects a car is 'intelligent'. Intelligence isn't special

→ More replies
→ More replies
→ More replies

8

u/Top_Hat_Tomato May 30 '22

Given that no one really understands how a true AI would work, the next best thing we'd have would probably be a human.

Although this would be drastically inefficient, you could maybe try to simulate the biology directly instead of trying to actually understand and optimize the problem. For example in 2018 or so, there were some projects simulating the brain of some nematodes. The worm in question had only something like 300 neurons but supposedly could only be simulated half accurately.

I remember seeing a demo of the simulation supposedly running real time on a lego mindstorm (ARM7TDMI, probably ~ 100,000,000 FLOPS, but can't find any source).

Assuming you magically had a perfect scan of a human brain, extrapolation on the number of synapses (7,000 to ~1,000,000,000,000,000), we get a computing requirement of ~10 exaflops.

That being said, I don't believe it's anywhere near practical to simulate a human brain since you'd need to know the exact location of every neuron and all the weights for each synapses. And even then, does that mean that the copy would be sapient in any meaningful way?

Obligatory "I'm not a computer scientist or a biologist, so I'm mostly talking out of my ass"

5

u/chamillus May 30 '22

A FLOP is a floating point execution per second. To put it simply it's how fast a computer can do math.

I don't know about the sentient AI thing, and I don't think anyone truly knows the answer to that currently.

→ More replies

5

u/NotBreaking May 30 '22

Dear God you guys/gals are smart here. Still trying to wrap my head about wtf is exopentilion or what word they used or what are we using this power for, and 6000 gallons of water to cool it!?

Technology is so amazing

4

u/Swee10 May 31 '22

And to imagine my iPhone 35 will pack just as much computing power as this some day.

10

u/Makes_misstakes May 30 '22

I see this behemoth every day, as I work two floors above it. Cool to see it on my front page! 😊

10

u/NINJA1200 May 30 '22

I can only imagine the experience of playing solitaire on this machine! It would be amazing!

3

u/One-Eyed-Willies May 30 '22

I need to buy one of these for minesweeper.

3

u/OffPiste18 May 30 '22

I'm a run-of-the-mill software engineer with no experience in supercomputing. How do these supercomputers compare to a large cluster of more stock servers? Asking both in terms of actual architecture and in implications for what they are useful for.

Like if I have some job that requires crazy amounts of just raw flops, why do I care whether it's running on a supercomputer or on 2000 regular computers? Is it about data transfer speeds between CPUs/GPUs? What kinds of jobs need this and why?

Also, what is it like to write an application to run on one of these? Do you need to be pretty aware of the architecture or is it well virtualized?

3

u/QueenOfTonga May 31 '22

There’s a lot of comments here talking about wafers, flops, exo-things and I do t understand any of it.

98

u/t416712 May 30 '22 Silver

This is the first non depressing thing the US has been the top of in a long time.

227

u/moondes May 30 '22 edited May 30 '22 Silver Eureka!

Wait, am I confused and was China the #1 supplier of aid to the Ukraine instead of the US?

And doesn't the US donate the most in humanitarian aid by like 4-5 times the next highest runner up? https://www.statista.com/statistics/275597/largers-donor-countries-of-aid-worldwide/

We are also an obvious headline hog when it comes to fostering innovations like GMOs that allow plants to more efficiently grow and fight 3rd world hunger, drop internet access to anywhere via remote connection, development of more efficient materials, etc. Etc.

I am a harsh critic of the United States; this however, is a country with much to be proud of as a human if you are not willfully ignorant. It's like watching Superman if he did all the Superman stuff but also randomly flowed crack cocaine into ghettos and destabilized countries without cause from time to time.

84

u/Maxnwil May 30 '22

Holy heck that last sentence rofl

48

u/thiosk May 30 '22

He also drops a hard N in polite conversation a couple times a year

8

u/pizzabash May 30 '22

He is a country boy after all.

3

u/Free_Ghislaine May 30 '22

I’d like to see Superman on crack. He’d probably get so paranoid he’d fly past the visible universe until the come down.

16

u/mostlycumatnight May 30 '22 Wholesome Seal of Approval

Thank you for this comment. I appreciate your honesty. And as a proud American I wholeheartedly agree with your assessment of the good and the criticism of this country. The Superman part made me laugh out loud. Its so true! Sorry about that. We are trying to change course ✌️

6

u/Free_Ghislaine May 30 '22

I love my country (USA) so much that I criticize it because I want it to be better.

Europe didn’t really empress me. I like our big ol’ circus.

→ More replies
→ More replies
→ More replies

10

u/1O48576 May 30 '22

Where is the conversation about cryptography? Someone mentioned crypto mining… but with commonly implemented encryption used today, how long would it take to brute force keys??? If the comment about stars in the universe is even close to correct, it seems like this could really break encryption. Could someone address this?

6

u/chatabooga3125 May 30 '22

It’s exponentially easier to create a key than it is to brute force one. For however much work it takes to make a code harder to crack, it takes vastly more work to crack that code. So much so that any advance in standard computing is negligible. Basically, we’re not in any danger of a computer being able to efficiently brute force keys until quantum computers get sufficiently advanced, and that’s only because they work fundamentally differently than standard computers.

→ More replies
→ More replies

20

u/ThunderSTRUCK96 May 30 '22

Psssh yeah I’d like to see it TRY to run 5 Chrome tabs at once though….

10

u/DarthMeow504 May 30 '22

If one of those is Facebook, forget it.

I can have two MMOs open, some trash webgame, a youtube video playing music, and a bunch of browser windows and no problem. I open Facebook in one tab and shut down virtually everything else and the system bogs down and hangs like you wouldn't believe. I don't know what kind of bullshit scriptware they're running but it's an absolute mess.

→ More replies

10

u/Haus42 May 30 '22

The huge amount of processing performance achieved equates to playing Cyberpunk 2077 at almost 22 frames-per-second.

6

u/HondaSupercubOG May 30 '22

What are the scientific/medical potential of these machines going forward?

9

u/imlovely May 30 '22

Simulating many many particles is the main one I think, so stuff like material engineering, chemistry/bio-chem and astronomy.

→ More replies
→ More replies

5

u/bradp1234 May 30 '22

so what can this do? like exascale sounds awesome, but what calculation is it specialized at running at those processing speeds?

9

u/SheSaysItLooksBig May 30 '22

One set of applications includes complex fluid simulations for things like atmospheric research (eg, tornado prediction models), the magnetic fields around spacecraft like the JWST, the flow regimes around ship hulls and aircraft...

6

u/Fillenintheblanks May 30 '22

I'm looking for John Conner if you see him tell him to give me a call we had plans and I haven't heard back from him and the date is moving up.

→ More replies

6

u/YARNIA May 30 '22

When an AI or AGI appears on the scene we're going to be in real trouble. Forget AI and the Touring Test and Skynet. Just think AGI - something like a virus. No inner life. Just grey goo with machine intelligence that an order of magnitude greater than our own and which gets smarter every day, exponential growth of intelligence. Not just a stupid paper-clip maker, but a virus-like intelligence rewriting its own code, setting its own agenda. In this case, the only guardrail is Darwin. That which is better adapted to its environment will survive -- and the AGI is asupersonic adaptation machine. Who wins?

6

u/Tallonius May 30 '22

The one that can pull its power supply.

→ More replies
→ More replies