Sounds like something which could be approximated by a DCT (discrete cosine transform). JPEG compression does this, and there are hardware accelerations for it.
> can make use of fast matmul acceleration
Maybe not, but matmul acceleration was done in hardware because it's useful for some problems (graphics initially).
So if these per weight activations functions really work, people will be quick to figure out how to run them in hardware.
Sounds like something which could be approximated by a DCT (discrete cosine transform). JPEG compression does this, and there are hardware accelerations for it.
> can make use of fast matmul acceleration
Maybe not, but matmul acceleration was done in hardware because it's useful for some problems (graphics initially).
So if these per weight activations functions really work, people will be quick to figure out how to run them in hardware.