Annoyingly GIF isn't supported as a source in <video> either, which adds extra complexity when you want to provide a proper fallback.
That being said, video in CSS background is a legit feature request.
You can deal with that server-side by checking referrers and other headers and serving HTML.