It continues to surprise me that the official documentation seems to prefer using global configuration in /etc/ansible; it always seemed vastly preferable to me to put it in a repository under my home directory. This does mean that I have to put in a config file to point at my inventory, but that's a trivial one time change and now I don't have to touch system configuration owned by root every time I want to change my inventory or whatever.
It seems like a lot of things are like that in Ansible. It seems like the preferred way to use it would be with a dynamic inventory, but most of the examples you'll find will have it hardcoded. Or you'd think you would to install it in a virtual environment or docker to deploy it in an isolated environment, but that is not recommended or even super easy to do. It sometimes feels more like you are tricking it into helping you.
Ansible ships ansible.conf and most (all?) distributions will store that in /etc. It's trivial to change to override this by having ansible.cfg in your project's folder.
Yes, I know; that's what I do. But it seems like a very strange default way to handle it given that it includes things like your inventory which are very much specific to any individual project. It would be like if Firefox shipped with its default profile location in /var or /usr/share and suggested that if you really wanted to then you could use a local configuration to move your bookmarks into your home directory.
For default/mandatory options, sure, but for inventory? Roles? Plays? I completely get having a sane default /etc/ansible/ansible.cfg, but sticking your inventory under /etc seems like a terrible mismatch. I guess to me, it's that ansible {inventory,roles,playbooks} aren't "configuration" files so much as they are, in a sense, "documents", and as such they should live in my home directory like all other "documents".
I think the intent is to reduce the friction of people who are used to Puppet where all of your code lives in /etc/puppetlabs/code on the master.
It's not an uncommon workflow to have a central "Ansible server" with a service account (hence why Tower exists) that's essentially used as a privileged bastion host in environments where sysadmin laptops don't have a way to reach servers directly.
I never get guides like this. This is a configuration management tool, and they're telling people to manually bootstrap their systems to use it. Great, you've helped someone do the thing the tool is intended not to do.
First of all, no, don't create an ansible user. Never use generic shared accounts. I mean, you can, but you can also just SSH into root. (You know what the difference between using the -B option and SSHing into root are? There is none.)
Second, either use sssd to enable an SSO system, or bootstrap all your hosts with an SSH CA so you don't have to constantly run jobs to add SSH public keys to all your nodes, which when you eventually start using ephemeral nodes, won't work at all unless it's using a shared account; see above paragraph.
Third, if you're going to use a plaintext inventory, generate it on the fly, and definitely don't write it to a single global root-owned file, because you may want to update it some time, and you may be running ansible more than once on that host at that time.
Fourth, they didn't really go over the anguish of trying to manage system-installed python packages and running Ansible. You need to install and run Ansible in a virtualenv with pinned versions, don't use a system-packaged version.
Five, don't use any complicated logic in your tasks or playbooks, or later you'll be kicking yourself because of how much of a huge pain in the ass it is to debug playbooks and tasks and inventories and filters and the non-standard fake YAML config format and a million other things. Keep it really really simple.
Six, run ansible-pull on a cron job with random jitter every 10-20 minutes, because your system and code will deviate over time, and you won't know it until you try to run Ansible again after 3 months and everything breaks. You'll also want a system to alert you when Ansible fails so you can find out what broke.
Seven, try to make any Ansible you use obsolete by baking Docker images, VM images, system packages, and other versioned immutable infrastructure artifacts. If you start out with that in mind, you may not ever have to touch Ansible once, and may even eventually achieve immutable infrastructure, and what a wonderful world that is. (I'm serious... everyone should be working to remove configuration management from their stack wherever it's feasible)
Good advice. Far better than the Red Hat branded info presented here. Personally, I prefer to push rather than pull, and I also use mitogen as the ssh strategy. It's incredible how much one line in requirements.in and another in ansible.cfg can speed things up.
We use Ansible to orchestrate our infrastructure and application deployment. Only in one particular stack are we using ansible how it is "supposed" to be used (SSH:ing into a machine and updating/installing Prometheus).
The infrastructure is provisioned using 90% CloudFormation and the other 10% is Ansible commands for when CF is not enough or when AWS CLI commands allows for better flexibility (i.e. generating SSH keys for EC2 machines and storing the private key in a central store). We make heavy use of the `cloudformation` Ansible-module.
Are you using Ansible to orchestrate the setup of a particular EC2 instance, or to actually provision AWS resources, or both? If you're doing AWS resources, where is ansible running from?
I have a few deployments using CloudFormation, and in those use CloudFormation::Init [1] to actually setup the EC2 instances (deploy packages, apps, configure local firewall, etc). This kind of grew over time and now involves some (mildly) complex bash and/or powershell scripts which do things that could probably be done with Ansible, but when I started it wasn't obvious how complex that part would eventually get. The split of Ansible vs CloudFormation responsibility also wasn't clear to me, and I favored the simplicity of a single tool, so even though I considered using Ansible, I ended up sticking to CloudFormation::Init.
In retrospect, CloudFormation::Init is pretty much a stand-alone tool, and the only connection it has to AWS and AWS CloudFormation is really just that it reads the metadata from the EC2 API to get its actual configuration (which is just a JSON or YAML blob).
We just use ansible for the provisioning of EC2 instances, and Terraform for the actual orchestration of the resources. Userdata is used to call ansible-pull. Theoretically we can use Ansible for everything, but Terraform's planning makes the runs a lot safer, and in general it just seems a little nicer for orchestration.
No, we are running Ansible Playbooks on the localhost. We store configurations in group_vars and have differnet environments vars which can override the global defaults ones (e.g. instnaces types) (when running the playbook one needs to provide the env, e.g. `-e env=dev`), we then give these varialbes to the CF stacks as parameters. We also use Ansible Vault to store sensitive information in Git, and push these to e.g. AWS SSM Parameter Store as encrypted SecureStrings to provide the software with secrets (we provide the apps with the name of the SSM parameter key)
EDIT: To clarify: the ansible-commands runs locally on your machine, or locally on the CI/CD agent - not on any remote machine/EC2 instance.
I'm looking to do something similar - spin up EC2 instances, deploy a script to them, run that script with various parameters, have it all defined in some template somewhere in our repo.
Did you look at EC2 UserData at all when you decided to use Ansible? I asked about this use case on the AWS Slack and someone pointed me at it. I haven't had a look yet but if you've used it as well as Ansible then I'd be curious to know how they compare/how you ended up with Ansible + CF instead.
I used Ansible in pull mode (https://docs.ansible.com/ansible/latest/cli/ansible-pull.htm...) in my UserData for several years. I've since left that company, but the implementation still powers their infrastructure. I had Ansible emit slack notifications when it started and completed. If the initial deploy ever failed, the node would eventually fail a healthcheck and that server would be destroyed and throw a different slack notification via an autoscaling group lifecycle monitor.
We use the userdata to call ansible-pull so that we can just terminate and re-create instances when we want to re-provision them. We use Terraform for the actual orchestration of the cloud environment.
That's the way to handle it. Use terraform to describe the shape and where the holes are, and use something like ansible to configure/provision the resource after it comes up and fits into the hole.
I just finished setting up Ansible to manage a bunch of Windows laptops. I basically just followed the example of Dag Wieers here[1][2], except that additionally every laptop is roaming with unstable IP addresses. So I used PageKite[3] to deal with that, although it would have been much easier if ansible-pull worked on Windows.
I know every experienced Windows admin would probably look poorly on my setup, but it feels really good to finally have some level of automation in managing these machines. It took me years to figure out a sane strategy under constraints like: OEM and retail software licenses only, employees retain admin privileges, no corporate network or VPN, unstable IP addresses, and (perhaps most importantly) no budget.
With constraints like that, it would also be a pretty good setup if you provide tech support to family and friends, as long as they trust you to backdoor their laptop.
We use ansible-pull to decentralize our Ansible runs as much as possible, but we also use Hashicorp Vault's SSH Key Certificates functionality to make sure our SSH keys are short-lived when we run it from CI. The last thing we want is CI with a bunch of root level keys on it!
you seem to be a big proponent of ansible-pull. are any of your use cases/implementations publicly available, I'm really interested to see how people are using ansible-pull in production.
I'd like to do something similar, currently we use a mix of ansible tower (which I don't love) and ansible runs from local machines to manage the infrastructure. I'd rather it all be tied into terraform though, so that we have a single place to manage changes from
We don't have anything publicly available unfortunately, but we call ansible-pull the instance userdata to configure the host on startup. IAM Policies and Vault integration are used to grant the host access to certain secrets needed by the ansible run.
Also interested in this, we have a fairly large setup and always interesting to see how others use it.
Especially around Vault, it never really clicked for me, I fail to see how it helps most of the times, if a user can trigger X that will generate a key on the fly, what prevents a hacker from doing the same, etc...
You can audit access/key-genesis, easily revoke keys, have stricter rules for giving out keys.
It's a complexity-compartmentalization trade off, that is usually recommended for better IT sec posture. Allows other stuff [IDS - intrusion detection system] to be built on top more easily.
Basically leads to the secrets in the secret vault mentality, so any time you see a secret not in the vault, you can sound the alarm.
In my corporate environment, I've played a bit with Puppet. Ansible looks to cover much of the same ground. Are there any compelling reasons to use one over the other?
THey have slightly different sweet spots, puppet is really good for fixing configuration drift, run it every 30 mins and you know the server will always be in the state you expect.
Ansible tends to be better for jobs you do when a sysadmin wants them done, its really good for scripting repeatable installations, or for doing updates in a consistent fashion on hundreds of machines, adding new users to a bunch of different applications, etc etc.
You can use puppet for ad-hoc jobs, but its not really designed for that it tends to be overly complicated and a bit awkward (imho). You can do drift re-mediation with ansible and run it every 30 mins to fix issues and it will work but its not its real sweet spot.
Personally I would use ansible for both use cases because its simpler, I understand it, and it has a more sysadmin way of thinking, but I know places that use both and that makes sense as well.
You may find Puppet obtuse if all you want is one time provisioning of new hosts. Sometimes you do care about procedural concerns like execution order, and in Puppet you have some influence (e.g. dependency relationships) but not direct control.
You may find Ansible backs you into corners with long lived mutable infrastructure. Configurations may drift, and your ability to run a playbook may depend on the sequence of playbooks previously run on the target. Ansible doesn’t really have a notion of converging on a goal state; it’s closer to scripting. You can have a single idempotent playbook that you run every time, but it’s not opinionated about that.
I'm deeply experienced with Chef and have been using Ansible for some newer projects.
Plusses:
- Supported Ansible modules are usually more reliable and more clearly defined in terms of how they'll function than community cookbooks for Chef are
- Upgrades between versions go smoothly
- People can just go read your YAML and figure out what it's going to do
Minuses:
- It's not code, so if you want to do something more programmatic, you're going to end up with something ugly compared to what you can do in Chef/Puppet's Ruby DSL
- It's still a bit immature in terms of comparison to Chef's server offering (Ansible Tower is pretty useless), and the Kitchen/Vagrant/Serverspec stuff is also not as advanced.
Ansible seems to fit well as a replacement for the Chef Zero use case of fire-and-forget provisioning - works well inside Packer or inside of Terraform via a null_resource, but I don't tend to find myself wanting to use it for long-lived machines that in the Chef world benefitted from continually running the cookbooks to help keep config in sync.
> It's still a bit immature in terms of comparison to Chef's server offering (Ansible Tower is pretty useless)
Meh, I really like Tower, but maybe that depends on what you're used to.
> It's not code, so if you want to do something more programmatic, you're going to end up with something ugly
this is true, ansibles DSL has its limits and the developers are pretty clear that there are some things that are too complicated for its yaml based syntax. 99% of use cases its great for, if you want that extra 1% then you have to write your own ansible modules (which is actually reasonably straightforward). I dont know chef but puppet has become more and more complicated over the years chasing the more advanced use cases and in many ways thats moved it away from a tool for sysadmins towards a being a tool for specialists so I think Ansibles outlook makes more sense. YMMV.
> I dont know chef but puppet has become more and more complicated over the years chasing the more advanced use cases and in many ways thats moved it away from a tool for sysadmins towards a being a tool for specialists so I think Ansibles outlook makes more sense. YMMV.
Yeah, I concur. There aren't too many cases where I can't work around the lack of programmability with some creative templates or what-have-you.
(I wonder if I could render an Ansible task file from a template and then include it...)
A big difference in our environment is that when puppet encounters a failure, it keeps going with everything that doesn't have an explicit dependecy on the failed resource. When ansible encounters a failure, it stops dead (of course, you can override that for specific tasks with "failed_when: false"). If you want the flexibility to choose what to fix next, you want puppet. In my arguably understaffed environment, it's useful to be able to say: "no, we can't do that yet, we must fix this first." Things end up cleaner.
They do different things. Puppet takes care of configuration drift, and makes sure that nothing you code into puppet changes.
Ansible is more like a readable bash on steroids. It's basically a way to distribute and execute Python code into other machines. Unless you have AWX (Tower), or create a custom repeatable delivery, it won't take care of config drift.
Overall, Ansible is much easier to write and understand. It's an extremely powerful tool. Puppet doesn't have any amazing advantages or disadvantages over Ansible (except of the simplicity, which is a large benefit for Ansible).
How do you test your playbooks targeting bare metals during development? So far I’m using docker but the official base image is very different from the official ISO, that’s kinda annoying (no ssh, no python, not even ps). I’m thinking about using Vagrant. I also saw a framework called Molecule but it’s based on docker so I guess it’ll be the same problem.
I just (in the last two weeks or so) started using Molecule. It took a bit to get everything set up and figured out but now that I've got things the way I want them it really is pretty awesome.
I'm using it with Vagrant and VirtualBox at the moment, just because that was the easiest way -- if you aren't already using Docker, that is -- to get started (due to available documentation, examples, etc.) but eventually I'd like to use both Docker and my ESXi lab for testing.
Testing w/ bare-metal was always a pain in the ass as I'd constantly need to "reset" the hosts to a specific "known good" starting point. I have everything set up for automated network installs, though, so it only takes 5 minutes or so to wipe and reinstall a physical host.
Now, I'm doing all my Ansible testing against VMs on VirtualBox, running from an image generated by Packer. Those images use the exact same preseed.cfg file I use on the bare-metal host, making it sooooo much easier. I do all my testing on the VMs, make sure everything is correct, and then run it against the physical host once for "verification".
Feel free to reach out (my username at Gmail) if (when) you run into issues or have any questions. I'm still learning new things as I go but I'm glad I decided to go down this path (and I'm happy to help if I can). It really is worth the time and effort!
