CHEYENNE, Wyoming — The National Center for Atmospheric Research (NCAR) is launching operations this month of one of the world’s most powerful and energy-efficient supercomputers, providing the nation with a major new tool to advance understanding of the atmospheric and related Earth system sciences.
There’s been some buzz in the last week as it was announced that NCAR’s new climate supercomputer was finally online. Back in 2010, I ran a story called “NCAR’s Dirty Little Secret” which outlined the new supercomputing center being built in Wyoming, while NCAR is in “green” Boulder Colorado. From that article:
“Having an NCAR supercomputing facility in Wyoming will be transformative for the University of Wyoming, will represent a significant step forward in the state’s economic development, and will provide exceptional opportunities for NCAR to make positive contributions to the educational infrastructure of an entire state,” says William Gern, the university’s vice president for research and economic development.
Gosh, what an opportunity for Wyoming. But why give the opportunity away? Colorado doesn’t want this opportunity? None of the politicians in Colorado want to be able to say to their constituents that they brought “economic development” and “positive contributions to the educational infrastructure of an entire state”? That doesn’t seem right.
The answer may very well lie in economics, but not the kind they mention in feel good press releases.
You see as we know from supercomputers, they need a lot of energy to operate. And because they operate in enclosed spaces, a lot of energy to keep them cooled so they don’t burn up from the waste heat they generate.
For all their sophistication, without power for operation and cooling, a supercomputer is just dead weight and space.
Electricity is king.
Interestingly, in the press releases and web pages, NCAR provides no answers (at least none that were easy to find) to how much electricity the new supercomputer might use for operation and cooling. They also provide no explanation as to why Colorado let this opportunity go to another state. I had to dig into NCAR’s interoffice staff notes to find the answer.
The answer is: electricity.
Measuring 108,000 square feet in total with 15,000-20,000 square feet of raised floor, it will be built for 8 megawatts of power, with 4-5 megawatts for computing and 3-4 for cooling.
8 megawatts! Yowza.
(I noted then that electricity is significantly cheaper in Wyoming.)
So besides the fact that NCAR abandoned “green” Colorado for it’s cheaper electricity rates and bond program, what’s the “dirty little secret?
Coal, the “dirtiest of fuels”, some say.
According to Sourcewatch, Wyoming is quite something when it comes to coal. Emphasis mine.
Wyoming is the nation’s highest coal producer, with over 400 million tons of coal produced in the state each year. In 2006, Wyoming’s coal production accounted for almost 40% of the nation’s coal. Currently Wyoming coal comes from four of the State’s ten major coal fields. The Powder River Coal Field has the largest production in the world – in 2007, it produced over 436 million short tons.
Wyoming coal is shipped to 35 other states. The coal is highly desirable because of its low sulfur levels. On average Wyoming coal contains 0.35 percent sulfur by weight, compared with 1.59 percent for Kentucky coal and 3 to 5 percent for other eastern coals. Although Wyoming coal may have less sulfur, it also a lower “heat rate” or fewer Btu’s of energy. On average Wyoming coal has 8600 Btu’s of energy per pound, while Eastern coal has heat rates of over 12,000 Btu’s per pound, meaning that plants have to burn 50 percent more Wyoming coal to equal the power output from Eastern coal.
Coal-fired power plants produce almost 95% of the electricity generated in Wyoming. Wyoming’s average retail price of electricity is 5.27 cents per kilowatt hour, the 2nd lowest rate in the nation
Named “Cheyenne,” the 5.34-petaflop system is capable of more than triple the amount of scientific computing performed by the previous NCAR supercomputer, Yellowstone. It also is three times more energy efficient.
Scientists across the country will use Cheyenne to study phenomena ranging from wildfires and seismic activity to gusts that generate power at wind farms. Their findings will lay the groundwork for better protecting society from natural disasters, lead to more detailed projections of seasonal and longer-term weather and climate variability and change, and improve weather and water forecasts that are needed by economic sectors from agriculture and energy to transportation and tourism.
“Cheyenne will help us advance the knowledge needed for saving lives, protecting property, and enabling U.S. businesses to better compete in the global marketplace,” said Antonio J. Busalacchi, president of the University Corporation for Atmospheric Research. “This system is turbocharging our science.”
UCAR manages NCAR on behalf of the National Science Foundation (NSF).
Cheyenne currently ranks as the 20th fastest supercomputer in the world and the fastest in the Mountain West, although such rankings change as new and more powerful machines begin operations. It is funded by NSF as well as by the state of Wyoming through an appropriation to the University of Wyoming.
Cheyenne is housed in the NCAR-Wyoming Supercomputing Center (NWSC), one of the nation’s premier supercomputing facilities for research. Since the NWSC opened in 2012, more than 2,200 scientists from more than 300 universities and federal labs have used its resources.
“Through our work at the NWSC, we have a better understanding of such important processes as surface and subsurface hydrology, physics of flow in reservoir rock, and weather modification and precipitation stimulation,” said William Gern, vice president of research and economic development at the University of Wyoming. “Importantly, we are also introducing Wyoming’s school-age students to the significance and power of computing.”
The NWSC is located in Cheyenne, and the name of the new system was chosen to honor the support the center has received from the people of that city. The name also commemorates the upcoming 150th anniversary of the city, which was founded in 1867 and named for the American Indian Cheyenne Nation.
Contour lines and isosurfaces provide valuable information about turbulence and aerodynamic drag in this visualization of air flow through the blades of a wind turbine, the product of a simulation on the NCAR-Wyoming Supercomputing Center’s Yellowstone system. (Image courtesy Dimitri Mavriplis, University of Wyoming.)
INCREASED POWER, GREATER EFFICIENCY
Cheyenne was built by Silicon Graphics International, or SGI (now part of Hewlett Packard Enterprise Co.), with DataDirect Networks (DDN) providing centralized file system and data storage components. Cheyenne is capable of 5.34 quadrillion calculations per second (5.34 petaflops, or floating point operations per second).
The new system has a peak computation rate of more than 3 billion calculations per second for every watt of energy consumed. That is three times more energy efficient than the Yellowstone supercomputer, which is also highly efficient.
The data storage system for Cheyenne provides an initial capacity of 20 petabytes, expandable to 40 petabytes with the addition of extra drives. The new DDN system also transfers data at the rate of 220 gigabytes per second, which is more than twice as fast as the previous file system’s rate of 90 gigabytes per second.
Cheyenne is the latest in a long and successful history of supercomputers supported by the NSF and NCAR to advance the atmospheric and related sciences.
“We’re excited to provide the research community with more supercomputing power,” said Anke Kamrath, interim director of NCAR’s Computational and Information Systems Laboratory, which oversees operations at the NWSC. “Scientists have access to increasingly large amounts of data about our planet. The enhanced capabilities of the NWSC will enable them to tackle problems that used to be out of reach and obtain results at far greater speeds than ever.”
MORE DETAILED PREDICTIONS
High-performance computers such as Cheyenne allow researchers to run increasingly detailed models that simulate complex events and predict how they might unfold in the future. With more supercomputing power, scientists can capture additional processes, run their models at a higher resolution, and conduct an ensemble of modeling runs that provide a fuller picture of the same time period.
“Providing next-generation supercomputing is vital to better understanding the Earth system that affects us all, ” said NCAR Director James W. Hurrell. “We’re delighted that this powerful resource is now available to the nation’s scientists, and we’re looking forward to new discoveries in climate, weather, space weather, renewable energy, and other critical areas of research.”
Some of the initial projects on Cheyenne include:
Long-range, seasonal to decadal forecasting: Several studies led by George Mason University, the University of Miami, and NCAR aim to improve prediction of weather patterns months to years in advance. Researchers will use Cheyenne’s capabilities to generate more comprehensive simulations of finer-scale processes in the ocean, atmosphere, and sea ice. This research will help scientists refine computer models for improved long-term predictions, including how year-to-year changes in Arctic sea ice extent may affect the likelihood of extreme weather events thousands of miles away.
Wind energy: Projecting electricity output at a wind farm is extraordinarily challenging as it involves predicting variable gusts and complex wind eddies at the height of turbines, which are hundreds of feet above the sensors used for weather forecasting. University of Wyoming researchers will use Cheyenne to simulate wind conditions on different scales, from across the continent down to the tiny space near a wind turbine blade, as well as the vibrations within an individual turbine itself. In addition, an NCAR-led project will create high-resolution, 3-D simulations of vertical and horizontal drafts to provide more information about winds over complex terrain. This type of research is critical as utilities seek to make wind farms as efficient as possible.
Space weather: Scientists are working to better understand solar disturbances that buffet Earth’s atmosphere and threaten the operation of satellites, communications, and power grids. New projects led by the University of Delaware and NCAR are using Cheyenne to gain more insight into how solar activity leads to damaging geomagnetic storms. The scientists plan to develop detailed simulations of the emergence of the magnetic field from the subsurface of the Sun into its atmosphere, as well as gain a three-dimensional view of plasma turbulence and magnetic reconnection in space that lead to plasma heating.
Extreme weather: One of the leading questions about climate change is how it could affect the frequency and severity of major storms and other types of severe weather. An NCAR-led project will explore how climate interacts with the land surface and hydrology over the United States, and how extreme weather events can be expected to change in the future. It will use advanced modeling approaches at high resolution (down to just a few miles) in ways that can help scientists configure future climate models to better simulate extreme events.
Climate engineering: To counter the effects of heat-trapping greenhouse gases, some experts have proposed artificially cooling the planet by injecting sulfates into the stratosphere, which would mimic the effects of a major volcanic eruption. But if society ever tried to engage in such climate engineering, or geoengineering, the results could alter the world’s climate in unintended ways. An NCAR-led project is using Cheyenne’s computing power to run an ensemble of climate engineering simulations to show how hypothetical sulfate injections could affect regional temperatures and precipitation.
Smoke and global climate: A study led by the University of Wyoming will look into emissions from wildfires and how they affect stratocumulus clouds over the southeastern Atlantic Ocean. This research is needed for a better understanding of the global climate system, as stratocumulus clouds, which cover 23 percent of Earth’s surface, play a key role in reflecting sunlight back into space. The work will help reveal the extent to which particles emitted during biomass burning influence cloud processes in ways that affect global temperatures.