Antares & FireAnt
Hardware Summary(totals):
ANTARES (the original system from 2002):
- 20 cpu's, type = AMD Athlon 1800+ MP (1.533 GHz)
-
7.5 GB DDR2100 RAM; disk space (IDE) = 640 GB,
- Interconnect: 100T (Fast Ethernet
for startup, administration, and a separate card to communicate with the
world) and 1000T or Gigabit Ethernet for computations*** (Intel NICs,
two DGS1008 switches from D-Link; cabling: normal twisted pair cat. 5e).
- 19" rack mounting in nine 2U units + 4U master. Own installation in a
rack using Tyan sliding rails. Computers put together by EUD Teknik,
switches bought & brought in a suitcase from the U.S.
FireAnt
(the 2005 upgrade more than tripling the computational power):
- 18 cpu's, type = AMD Opteron 248 (2.2 GHz)
-
18 GB DDR2700 RAM , disk space (SCSI) = 657 GB,
- Interconnect: 100T (Fast Ethernet
for administration) and (dual) Gigabit Ethernet for computations***
(3COM dual NICs on mobo, one DGS1016 switch from D-Link; cabling: UTP cat. 6).
- Nine rack-mountable 1U servers
Sun Fire V20z .
(See also Sun's page).
- Remote administration ("lights out") processors, allowing upgrade and/or restart via ssh over
the dedicated ethernet ports.
- SunMicro offers sliders
for rackmounting V20z's for 1100 kr/machine, which would set us back
10kSEK. We opted for a proprietary design costing 100 times less (100 SEK
total; actually 40 PLN in Warsaw, Poland).
Here is how it all looks in JPG format:
without
the servers , and here
with the servers, up front, and here
from the
back side (with crosses patch cables daisy-chaining the machines for
administration still missing).
Here
Ant and FireAnt are seen together. As usual, the older, slower, system
is bigger and looks more respectable. Notice elegant cabling too.
- 21 Jan 2005:
FireAnt is now fully physically installed. Awaiting the Frankenstein moment
(Igor! switch on!!!).
***(Our interconnect transfers up to 0.5 Gbit/s of data from one cpu
to another, only 2-2.5 times slower than the Dolphin Wulfkit3 network on
the #2 Swedish supercomputer Seth. Click to see the details of the
comparison
calculations)
Price/Performance:
ANTARES (2002): Maximum estim. performance = 61 GFlop, a factor ~2.2 away from qualifying
for the 2002 list of world's Top 500 Supercomputers.
While this itself isn't bad, the really neat thing is the price/perf. ratio,
which shows you how CHEAP we are :-)
Total cost of the system = 207 kSEK + VAT. (~$20k+tax at the time of
purchase; we don't
pay VAT at the University.) Can YOU beat our $330/Gflop ratio (in
rack mounting, which involves some extra cost)? Please let us know if you
do, but frankly we're not expecting your email before, say, late 2003.
[We didn't get any....]
FIRE ANT (2005): Maximum estim. performance = 144 GFlop, assuming the
61 GFlop number above, significantly
beating the Athlon system because of the ~2.5 times faster cpu's. While
Opterons are 64-bit machines, we'll mostly benefit, not from that but from
the on-the-chip memory controller (once again, memory bandwith is tied
to the cpu speed!).
Price/perf. ratio's quite good.
Total cost of the system = 202 kSEK + VAT. (~$30k+tax, of which we don't
pay tax. Notice how the exchange rate changed...) Can YOU beat our $206/Gflop ratio of the new Sunfire system?
Please let us know what ratios you were able to achieve.
Main Software in Use: Debian
Linux, MPICH, PVM, Absoft Fortran
Main Applications:
extrasolar
planet formation (CFD of protoplanetary disks),
circumstellar
dust disks
Slide show on original Antares (2002):
Historical Footnotes:
- Antares is a younger, 10 x more capable, brother of
Hydra.
. The $100 question is: will Antares be as reliable as Hydra? We are
a bit unsure (we have experienced some hardware problems (mobo's) already a few
months after inauguration )-:
- Note added in Sept 2003: stability OK this year, no reboot for
>0.5 yr. We added 2 120GB IDE disks recently.
-
Note added in Nov 2004: this year, two nodes died apparently due to
mobos, and one has a gigabit NIC failure but we reach it via Fast Ethernet
(so, after all, it IS utilized..). We are not very happy :-( with overall
stability. Plans are made for a new system or radical upgrade early next year.
Stay tuned. However, we must note that D-Link switches worked perfectly.
-
Note added in Dec 2004: FireAnts & switch & cables ordered. An alternative
system was considered made of: MSI 1U servers K1-1000 w/o memory or cpu
(available for 4400 kr each) plus Opterons 250 and 1 GB DDR RAM/machine.
While 25% cheaper, this solution provided no warranty vs. Sun's 3 yr warranty;
we also have a general warm feeling about Sun hardware (cf. remarks on Hydra).
Acknowledgments: Funding for Antares (earmarked 200 kSEK research grant)
and Fire Ant extension (202 kSEK from a general-purpose research grant) was
generously provided by Vetenskapsradet (Swedish Science Board). We also
acknowledge Iouri Belokopytov, Sergio Gelato, and Uno Wänn from the
Stockholm Center for Physics, Astronomy and Biotechnology for their interest
and help in building this system: the racks provided by Iouri, system work by
Sergio, help with the rails on Antares by Uno. Adam Peplinski spent much time
and effort brillintly running Antares as a sysadmin. Recently, a
grandfather-father-son team of Artymowicz's contributed the rack-mounting for
FireAnt: Mikolaj (purchasing, drilling holes), Pawel (design, installation),
Adam (installation, simulation of a percussion set using aluminum L-beams and
cardboard packaging of servers). Many thanks to all.
Moja odpowiedz na pytania wytrwalych tropicieli spiskow, komunistow, cyklistow,
zydow (koniecznie z malej litery) i innych nieprawdziwych Polakow: nie, nie trafiliscie zupelnie. Musicie szukac dalej. (Moj dziedek byl w Wermachcie.) Wytrwalosci i powodzenia zycze.
last update: Jan 2011, P.Art.
