solved. the documentation in this area is kinda weak. need the "unsupported GPUs" option, the "enable GSP" option is not enough by itself.
Everything with NVIDIA is a pain in the neck.
Wed May 03 2023 07:24:18 AM EDT from LoanShark
what the hell is going on with NVidia's proprietary driver.
Documented module parameters do not actually exist, so can't enable GSP mode on the closed-source kernel model and can't enable the Unsupported GPUs flag on the open driver.
miracle of miracles, I got nvidia hardware video decode working in firefox.
then it broke itself because I was apparently tracking firefox trunk...
that's fixed though. Works best on the -510 driver. Even though Wayland is an epic fail on that version.