Homelab upgrade to a EPYC system

I replaced my "consumer" server with "enterprise" parts to scratch the itch of building something with my hands after spending the majority of my time building software. This post details my journey and some fun lessons I learned.

Where I started

Like most homelabbers, I started with a Raspberry Pi. Somehow that escalated to a Unifi network system, and finally a server to run Proxmox on top of the AM4 platform.

However I couldn't use more than one GPU and I didn't have enough PCIe lanes for my 25Gbps NIC.

Plus, I was confined to my apartment for a bit due to a back injury, so I thought, why not make a change?

I figured I'd need a CPU with more PCIe lanes, and started doing research on the Threadripper and EPYC platforms.

I ended up finding a great deal on an EPYC 9654 CPU with 96 cores which led me into quite a rabbit hole.

The EPYC 9004 series processors use the SP5 socket. At the time of the build, there was already a lot of motherboards that had support for my CPU, but I also wanted to make sure that I could add several GPUs and maintain a 25Gbps network.

Server motherboards

Up until this project I wasn't really expecting to learn any more than I've already learned about hardware -- but I knew I was going to need to find a standalone motherboard and not buy a barebones server. It would've certainly been easier - but I heard that they are loud, which would be absolutely unacceptable in my very spacious NYC apartment.

Well, after ordering my first standalone server motherboard, I very quickly learned that there's a reason why most go with a barebones server and don't try to build a system from scratch.

ASRockRack BERGAMOD8-2L2T

This was the first motherboard I looked at. I couldn't quite understand what this motherboard was designed for. What's up with the PCIe x8 lanes at slot 0 and 2? I'm pretty sure there are enough PCIe lanes left to keep them at x16...

Gigabyte MZ33-AR1

I don't know how I missed it, but I got so excited by the RAM possibilities that I clearly missed a few things.

Those PCIe lanes look awfully close to the RAM slots...

Sigh.

ASRockRack GENOAD8X-2T/BCM

Finally, we get to the one I stuck out with.

This motherboard is kinda crazy. It has Phison PCIe signal repeaters all over the PCB to make it possible to drive 7 x16 slots.

Anyways, let's get into the gotchas I ran into while building my server.

Complications

NVMe

If you want to use use both NVMe slots, you can't:

• Use PCIE1 at x16. Sucks if you want to use a GPU that's 2 slots wide!
• Set the link width of PCIE1 to x8 to use a 25Gbps NIC. It has to be x4x4 otherwise it won't pick up both NVMe ports.

MCIO

Configuration gotchas:

• MCIO4 is PCIe only
• MCIO3 and MCIO4 don't work if you use the PCIE7 slot
• MCIO1 and MCIO2

Cable gotchas:

These are 4i interconnects - if you want to connect to a backplane, it's proabably using the SFF-8643 (internal mini SAS HD). There are no readily available MCIO 4i to internal mini SAS HD cables, but I was able to find one eventually on amazon.

Power

This motherboard requires three EPS connections to the PSU. There is literally only one ATX power supply that comes with three EPS cables, and that's the Seasonic Prime TX/PX 1600W PSU.

Full part list