Kata works pretty well. You set it as a runtime for docker and it gives a fantastic illusion that the container isn't different from any other docker container but uses a KVM VM with minimal hardware to run the container. It starts in under 200ms (and faster if you make it use firecracker), etc.
The problem your parent is pointing out is that AWS instances don't support nested virtualization (and it sounds like neither does Digital Ocean) so on both of these cloud providers you can't use something like kata containers or any nested VMs. GCP does support nested virtualization in any VM and aws `.metal` instance types do as well, but are rather expensive. It's really a shame that even with the new EC2 kvm-based hypervisor they still didn't enable nested virtualization for most instance types, otherwise we'd definitely be making heavy use of kata containers.
I wish Oracle hadn't bought Ravello Systems and that Ravello open sourced their binary translation stuff that made nested VMs possible in EC2 without the full overhead of software virtualization. Unfortunately, there are no open source implementations of similar software that I know of. Their blog is now hosted on oracle's site: https://blogs.oracle.com/ravello/nested-virtualization-with-...
We tried kata the user experience is not as good as lxd.
Moreover with lxd just use the same deployment platform, scripts as used for bare-metal and VM. No need to fiddle with mix of shell scripts and DSL for container orchestration and how to integrate with your own code. No need for zombie process like in docker.
Kata is great if you're already using docker, it also is increased isolation versus lxd.
If you don't already have a use case for docker, kubernetes, the image repository features, etc then of course kata wouldn't be useful to you over lxc.
Also, in case you're dealing with zombie processes in docker again and can't ditch docker, running a container with `docker run --init` gets you a minimal init process that reaps the zombies. Adding that flag to dockerd instead does this for all containers by default. It's insane that this isn't turned on by default since it's such a common problem. Additionally, if you want to run systemd inside the container similar to lxc, you can install the systemd-oci-hook and it will do the necessary setup for systemd to be happy.
Thanks for the information and indeed I looked at Kata again. But just it didn't click like the way LXD did. I found with LXD the image repository are so easy, deployment my team can re-use whatever they learned for bare-metal and virtual machines and last but not the least its performant with nice tooling.
Initially when I started using had few networking issues due to different support in GCP, AWS and Azure. But now my team is proficient enough to use it even on them. Also the constant updates Stephen and his team did is fantastic.
If you have time please look into weekly updates and play with LXD. It works great for reasonable sized container cluster which covers most of the startups.
They won't be good for big google size of kind yet without more tooling like Kubernetes.
The problem your parent is pointing out is that AWS instances don't support nested virtualization (and it sounds like neither does Digital Ocean) so on both of these cloud providers you can't use something like kata containers or any nested VMs. GCP does support nested virtualization in any VM and aws `.metal` instance types do as well, but are rather expensive. It's really a shame that even with the new EC2 kvm-based hypervisor they still didn't enable nested virtualization for most instance types, otherwise we'd definitely be making heavy use of kata containers.
I wish Oracle hadn't bought Ravello Systems and that Ravello open sourced their binary translation stuff that made nested VMs possible in EC2 without the full overhead of software virtualization. Unfortunately, there are no open source implementations of similar software that I know of. Their blog is now hosted on oracle's site: https://blogs.oracle.com/ravello/nested-virtualization-with-...