Setup Proxmox on TTY

By default, Proxmox 6.3.1 does not detect RTX3090. You need to explicitly add:

#/usr/share/X11/xorg.conf.d/nvidia.conf

Section "Device"
    Identifier "Card0"
    Driver "fbdev"
    BusID "your_busid"
EndSection

# Check busid
lspci -k | grep VGA     # looks like ##.##.#

Then run xinit to launch into Proxmox installer

Enable GPU Passthrough

In my testing, I get native performance with passthrough.

Enable IOMMU and interrupt remapping

# Enable IOMMU
echo 'GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt video=efifb:off nofb nomodeset video=vesafb:off"' >> /etc/default/grub

echo "quiet amd_iommu=on iommu=pt" >> /etc/kernel/cmdline

echo "vfio
vfio_iommu_type1
vfio_pci
vfio_virqd" >> /etc/modules

# Enable interrupt remapping
echo "options vfio_iommu_type1 allow_unsafe_interrupts=1" > /etc/modprobe.d/iommu_unsafe_interrupts.conf
echo "options kvm ignore_msrs=1" > /etc/modprobe.d/kvm.conf

Add device passthrough to VFIO

VFIO_IDS=$(
    lspci | grep NVIDIA |  # Get NVIDIA PCI devices 
    cut -c 1-7          |  # Get just the bus number
    xargs -n1 lspci -ns |  # For each device:
    cut -c 15-23        |  #     get the vendor ID code
    paste -sd ","          # Combine into CSV format
)

echo "options vfio-pci ids=$VFIO_IDS disable_vga=1" >> /etc/modprobe.d/vfio.conf

Blacklist GPU drivers (so VM can use them)

echo "blacklist radeon
blacklist nouveau
blacklist nvidia" >> /etc/modprobe.d/blacklist.conf

Update Grub and reboot

pve-efiboot-tool refresh
update-initramfs -u -k all
update-grub
reboot

Check that everything is working

dmesg | grep -e DMAR -e IOMMU - e AMD-Vi

[    0.592815] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported
[    0.594154] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x80
[    0.594155] pci 0000:00:00.2: AMD-Vi: Extended features (0x69a22da13324bcf):
[    0.594156] AMD-Vi: Interrupt remapping enabled
[    0.594157] AMD-Vi: Virtual APIC enabled
[    0.594157] AMD-Vi: X2APIC enabled
[    0.594240] AMD-Vi: Lazy IO/TLB flushing enabled
[    0.594844] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4 counters/bank).

dmesg | grep -i vfio

[    5.953629] VFIO - User Level meta-driver version: 0.3
[    5.957169] vfio-pci 0000:0e:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=io+mem
[    5.980174] vfio_pci: add [12ed:3304[ffffffff:ffffffff]] class 0x000000/00000000
[    6.000165] vfio_pci: add [12ed:2bfa[ffffffff:ffffffff]] class 0x000000/00000000
[   34.528192] vfio-pci 0000:0e:00.0: vfio_ecap_init: hiding ecap 0x1f@0x260
[   34.528213] vfio-pci 0000:0e:00.0: vfio_ecap_init: hiding ecap 0x20@0x800
[   34.528221] vfio-pci 0000:0e:00.0: vfio_ecap_init: hiding ecap 0x27@0xd2f
[   34.528222] vfio-pci 0000:0e:00.0: vfio_ecap_init: hiding ecap 0x28@0xe00
[   34.528223] vfio-pci 0000:0e:00.0: vfio_ecap_init: hiding ecap 0x26@0xf00
[   34.529465] vfio-pci 0000:0e:00.0: No more image in the PCI ROM
[   34.548134] vfio-pci 0000:0e:00.1: vfio_ecap_init: hiding ecap 0x26@0x170
[   36.127601] vfio-pci 0000:0e:00.0: No more image in the PCI ROM
[   36.127623] vfio-pci 0000:0e:00.0: No more image in the PCI ROM

cat /proc/cmdline

BOOT_IMAGE=/boot/vmlinuz-5.4.73-1-pve root=/dev/mapper/pve-root ro quiet amd_iommu=on iommu=pt video=efifb:off nofb nomodeset video=vesafb:off

For each VM with GPU passthrough required

_DEVICE=$(lspci | grep VGA | head -c 5)
VID=100  # your VM ID here

echo "machine: q35
cpu: host,hidden=1,flags=+pcid
numa: 1
hostpci0: ${_DEVICE},pcie=1,x-vga=1
bios: ovmf
scsihw: virtio-scsi-pci
vga: virtio
args: -cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,hv_vendor_id=NV43FIX,kvm=off'
" >> /etc/pve/qemu-server/$VID.conf

Now you can start the VM with full GPU passthrough.

If you have multiple devices, you can replace $_DEVICE with the specific device you want to passthrough

Create ZFS Pools

Detect drives

fdisk -l | grep '^Disk'

Format drives

If creating a fresh pool, delete everything on the drive with d

fdisk /dev/your_drive

Setup zpool

zpool create -m /mnt/target_name target_name raidz_type /dev/drive1 /dev/drive2 # ...

Create Backup Server

Install Proxmox Backup Server

Download and install the PBS ISO
Install as a VM on current Proxmox Server (if backups are not critical)
Install on a separate machine off-site (if backups are critical)

Add Storage to PBS

In PVE: Add new SCSI device to PBS VM with target disk size (500GB should be enough to get started). This can come from your ZFS pool.
In PBS: Drives should appear mounted. Initialize these disks with GPT (Administration > Storage/Disks). If coming from your ZFS pool in (1), you don't need to create another pool in PBS.
In PVE: Add new PBS storage (Datacenter > Add > Proxmox Backup Server) with the same name as from (2) and enter SHA-256 fingerprint from PBS (Dashboard > Show Fingerprint)

Schedule backups

In PVE: Create backup jobs (Datacenter > Backup > Add) with preferred settings. Defaults are fine.
In PBS: Configure how many backups you want to keep. Keeping a few dailies, 1 weekly, 1 monthly is fine. Keep longer for critical applications.
In PBS: Setup verify jobs (Datastore > Verify Jobs). Defaults are fine. This will validate your jobs for corruption.

Proxmox (QEMU/KVM/VirtIO)