One of the authors here. This website is still a work in progress.
It's an ongoing effort to gather all the resources that will help you use Erlang in a business. The booksite is divided in three sections focusing particularly on Erlang/OTP’s higher level concepts in the current open source ecosystem, how to use it in production (while setting up a pipeline for continuous development and delivery), and how to build a team when you’re starting from scratch.
We're working mostly linearly in each section, but they're not all progressing at the same pace. We have nothing yet on "building a team" although our outlines are all planned; the "hard things to get right" chapter was supposed to come in later but we saw enough questions about these topics to rush it out.
We might release things from each section as they are ready and we see fit. In the meanwhile, we're always appreciating feedback and people checking in to see what's new in terms of content.
One of the things I've learned from many attempts to document internal processes at organizations, is that often the authors are least equipped to explain things.
They know too much, they often don't recognize how many pieces of assumed information they're injecting into their so-called explanations. You often get circular explanations in addition to circular reasoning and it can be off-putting.
A better use of time is to get someone who is just started to exhibit competence to write the docs. They won't assume the jargon, they can't assume deep knowledge, and because they're living right at the pain point they can be especially sympathetic. And if what they explain is dead wrong, this is your chance to correct a horrible misunderstanding before they get too far along. Pointing out the 'false friends' in a system early on can be pretty powerful. Educationally or as an opportunity for improving the system.
In other words, I'd encourage you to invite new people to contribute. Just keep an eye on what they do and be ready to jump in with corrections.
There's such as thing as too verbose, but it usually comes from writing stuff in the wrong kind of document: there is always a place for in-depth explanation, and there is a place for terseness to the point of familiarity.
One last thing: if your topic has a jargon of its own, a particular meaning to words, introduce these as early on as possible, and be excellent in consistency throughout docs (in addition, consider a definition link/info widget for each such term on its first occurence in any document, letting people 'jump right in' to any section/article). If you are going to force people to re-wire their brain to suit your particular vocabulary, you should maintain 'immersion' at all times.
If find this especially true in not-so-well-defined paradigms such as OOP when approached by various programming languages. You need a "map of terminology" more than anything else to quickly grasp how to use language XYZ.
> You need a "map of terminology" more than anything else to quickly grasp how to use language XYZ.
Exactly why I hate Brew with passion. In the doc you find: formula, bottle, cask instead of package and... no idea what the others mean. I have no time to learn and remember the mapping, which exists for no other reason than the author trying to be cool.
Totally agreed! I think Brew will eventually be remembered as "this bad way of entering the topic of interest", like BASIC for programming, or a learning js / web programming through a framework first.
It's teaching you the wrong way, making you almost illiterate about pm when you enter the real thing (here Linux/UNIX world, or Python after BASIC, vanilla web concepts, etc.)
Anything that helps 'lateral' access to a knowledge base is helpful, because you don't need to go through all the bottom-up (learning) or top-down (prior knowledge) to explore and reach a given problem space. You can simply 'cheat' directly to 'stage N'.
Once there, you might be missing key information: principles above you, terminology and base config below; this is where a glossary (hyperlinked automatically ideally) is a life-saver. Follow 1-2-3 links and you've got the gist, back to 'stage N' to the next step, rinse and repeat.
When documentation is badly done, this approach may result in endless circular references, feeling 'trapped' at or around 'stage N' — you explore that stage all you want but gain no understanding of how it fits into the whole. But when it's well executed, such documentation is a treat — RHEL comes to mind as being very practical to use in that regard, for a rather complex/deep doc base.
Yes and no. Some of the struggles are fresher than others, and in some cases you do internalize things to the point they feel like a second nature.
The effort mentioned here of getting newer people to read the content and/or produce it is something I've done a lot in the past, and one of the ways I've used in on-boarding people as well. We (the co-authors of Adopting Erlang) tend to do it with topics where we don't know everything the other authors know, and provide feedback to each other about things that lack clarity.
I've also taught Erlang stuff in multiple places and started knowing some of the things people find confusing as a kind of pattern, which may get easier to address early on with time.
The thing that can never be done well, though, is predicting the things someone with an entirely different background from yours would find confusing; the context carried by their experience influences the things they understand and the information they extract from a text. Teaching to a functional programmer is different from teaching to a developer who has done OO their entire career. Teaching to someone doing front-end work is going to be a different experience from teaching to someone doing mobile or back-end work. The same may often extend to sociological, economical, or cultural factors as well.
All of this means that each teaching experience has new surprises as well and you can't just cover them by thinking very hard. Frequent reviews, both technical and from editors can help, though, along with your own experience when teaching.
No. What you're describing is the problem people have when teaching/providing insight/responding to questions online. They all use their current understanding and apply it to the question not assuming the other person has no knowledge of the subject.
Haven't you ever seen someone ask a question online that had a "newbie" answer that is generally correct, only to have a ton of people flood your response with all the caveats ignoring who their audience actually is. The default for most people is to use your current knowledge.
Becoming familiar with a system makes you forget what was and what was not obvious. I experience this with myself. Once I've learned something well, then I can find it hard to explain it to someone else. I find it best if I write down assumptions that got corrected or unclear things as I encounter them.
I've seen this in other disciplines, and if anything to worse degree. I've witnessed situations where the old masters can't talk to the journeymen, and the new masters can barely talk to the beginners. You learn by watching and whatever the people a couple years ahead of you can decipher for you.
To make matters worse, what often happens is that the instructors believe they have taught something, and they don't witness the actual learning take place, so they continue in their belief in their practices.
As someone who did a lot of professional training and love to teach complete beginners it’s definitely possible but it’s not intuitive and it’s a lot of conscious work.
I think they deserve more thorough coverage because 1) you really need them in a lot of applications, and 2) they're not in OTP, so picking one (especially if you're not really an Erlang expert) and employing it in the right way are a bit trickier.
A simple example of an application that probably needs a circuit breaker is a web application that can keep running even if the database is not available, even if all it does is return an error message to the browser.
Don't circuit breakers arise naturally from configuring supervisors in a particular way? I.e. supervisor failover after a certain number/period of retries.
Just wanted to thank you and Tristan for the great resource. It's much appreciated!
The cheatsheet about what the processes do when they trap exits/are linked/are port of OTP is awesome. I never remember that very well and was looking for a handy reference to have. Now I have it.
Docker and Kubernetes references are great as those are often involved in today's development and operations, and everyone is probably wondering what's the best way to combine those with Erlang.
There is a large overlap with Elixir stuff, mostly at a higher level; whatever covers OTP, handling supervision, releases, docker image practices, approaches to testing, production practices, etc. will very likely be useful to Elixir users.
But the lower-level details like handling dependencies, build tool commands, test framework specifics, the code snippets and samples, and so on are all going to be Erlang-specific.
I use both languages frequently for work and there's far more in common between them than there are differences. If that's not your cup of tea and Elixir is really all you're after, you can look for Ben Marx's Adopting Elixir (https://pragprog.com/book/tvmelixir/adopting-elixir) book. We are not affiliated with it and don't aim to cover the same content (we just have similar titles), but it's a good option that goes Elixir-first.
Anyone who has written a technical book knows that in general they cost a lot in terms of time and don't generate great returns. The best case is probably if it boosts your consulting income or something to that effect.
I've never met Fred, but I feel his book "Learn You Some Erlang" is clearly in another category: it's a "labor of love". I can't speak to his precise motivations, but the book is so rich and thorough, that it's clearly something he worked way harder on than would have been strictly necessary just to whip out a quick book establishing himself as an Erlang Expert. Maybe it was "the Erlang book he wished he'd had when he got started"?
As such, it's one of the few tech books I've bought, and done so quite happily, in recent years.
Well this is technically the fourth one I'm working on. All are available for free online because I want these resources to be accessible first and foremost.
I've got:
Learn You Some Erlang, which has been published with No Starch Press. This was my first one, which I assume is the one you're talking about. Lots of drawings, something I no longer really have the time and energy for these days compared to early in my career. It's the big basics of Erlang. A bit dated now, particularly since the ecosystem evolved around it, but all the basics are still well-covered I think.
Erlang in Anger, only available as a self-published short e-book available for free, meant to teach people to operate Erlang in production.
Property-Based Testing with PropEr, Erlang, and Elixir; my latest published one (with Pragmatic programmers), covering property-based testing both with Erlang and Elixir at once. Pragprog have let me keep the initial website online, which only covered Erlang.
And Adopting Erlang is the latest one. We're still writing it (this time I'm a co-author rather than the sole author), and there's no physical or ISBN'd copy of it available at this point in time.
All of them have to be labours of love, just because it's a lot of work and effort, and at this point in my career, my name's well-known within the Erlang and Elixir communities, so you could say there are diminishing returns. I still enjoy writing though, it's just harder to make time.
thank you for the answer. I'm not too interested in erlang (I'm still learning elixir so 1 horse at a time) but I'll keep your book close since both overlap quite a bit and the OTP part is common to the two languages. Thank you for writing that book !
What are the arguments for adopting Erlang instead of Elixir for new projects in 2019? I understand there is a lot of existing Erlang code out there, but it has a reputation for being less approachable than something like Elixir (syntax and tooling).
Erlang's tooling has vastly improved in the last few years to the point it's not as much of a concern anymore (a formatter is the big missing thing), and I just happen to prefer its syntax to Elixir's in the first place.
While I can argue in favor of tooling being roughly on-par (sometimes better, sometimes worse), I can't really discuss syntax since this is often just related to taste more than anything.
My stronger argument would usually be in favor of the community you end up with. While both communities are starting to merge and grow closer over time, your average Erlang dev is likely to work on infrastructure components, and your average Elixir dev is likely to be working on web apps. I personally like working on the former better, so there's more limited interest for me in terms of most Elixir jobs anyway, even if it's not like no infra work exists in Elixir.
Both can have a great time working on APIs of any kind, and both can reuse most libraries of each other's community, so as I said, this is starting to become a moot point though. I personally have no problem with either.
In what cases do you see Erlang's tooking as better than Elixir? I have generally found Elixir's tooling more approachable, but I haven't yet had to do anything difficult.
I'm obviously biased since I'm one of the Rebar3 maintainers but: I prefer a declarative approach to config files and whatnot; composable profiles are really neat; the plugin system for reusable tasks; dialyzer "just works" with it; the ability to compile and build projects in non-elixir BEAM languages.
That being said, Mix does have its advantages in other ways (easier to extend for one-off scripts, for example, since it won't actually need a whole plugin; the ability for mixed-language projects with Elixir and Erlang), and they're clearly the reason (with Hex) why we have good package management today.
Again, I'm not ready to say one language has better tooling than the other; it's just that I feel it's a far cry from the situation from a few years ago where Elixir was just miles ahead -- the gap has closed since then.
I wrote the Dialyzer pretty printer for Elixir, and I had to jump through myriad hoops to get that to play nicely, and I still do not have it 100% working in all cases yet, though there's way fewer bugs reported these days.
I would argue, probably without a great deal of agreement, that Erlang’s syntax is simpler than Elixir. It’s less familiar, but there’s practically nothing to it.
A section on "stuff that's not in OTP, but many people end up using".
For instance, poolboy comes to mind, alongside the circuit breakers I mentioned in the other comment.
lager for logging.
At one point, there was a lot of stuff in the basho github repo that seemed pretty widely used.
It's been a bit since I've used Erlang so maybe there are some others that come to mind as "you're likely to see this included in a production Erlang system".
As far as I know, yes. Elixir sends its own internal logs through it, and some of the Erlang libraries have begun making it optional recently. I don't include lager on purpose in any of my Elixir projects, for what it's worth.
You could and probably should promote this to the Elixir community too. Especially the OTP parts are very relevant fundamental knowledge for any Elixir dev.
This might be on your radar already, but the rendering is broken in Firefox. I'm using version 60.8.0esr, which is what I get with "apt-get install firefox".
When I open the link I see the table of contents on the left and a bunch of empty white space. Scrolling down, the content eventually comes into view, but scrolls on top of the table of contents, making it impossible to click.
I'm blocking the google fonts site, so maybe that's part of the problem.
There seems to be a slight encoding issue with external URLs. In the dependencies section under "Private Hex Mirrors", there is a link to mini_repo which results in a 404. It seems like the underscore in mini_repo is encoded to %5f which github doesn't seem to like?
This is amazing, thanks for all of your hard work! It's very encouraging to see continual effort put into the Erlang ecosystem. I haven't used Erlang in around 5 years so this looks right up my alley for getting up to date with the latest best practices.
As an aside, it's been a little bittersweet seeing all of the attention and momentum going into Elixir. Yes it's great for BEAM and Erlang certainly benefits from those contributions as well. But there's just something so beautiful about Erlang, it deserves more attention than it gets. Thanks for keeping the dream alive!
As an Elixir dev for coming up on four (four! Jeez where has the time gone?) years, I’ve yet to really delve into Erlang. I can read the syntax well enough to figure out why my Hackney request is doing odd things with my HTTP headers etc..., but that’s about it. To all Erlangers out there, what advantages/niceties/powers do you have in Erlang that make you reach for the language over Elixir?
Mostly I just like it better. But otherwise in my day-to-day off the top of my head:
Less “magic” (fewer macros that may change semantics, straightforward config handling), focus on declarativeness, no variable re-binding, more powerful test framework (common_test), seamless Dialyzer support, better logging framework for structured logging, simpler/minimal syntax, personal preferences for tooling (until recently, direct support for releases was lacking in Elixir), and force of habit.
I've noticed there's a section in there about deploying with Docker. I know many other development communities swear by Docker, but a number of the OTP/BEAM devs I've spoken to seem unsure about it.
I also tend to believe Docker makes little sense, where OTP releases are the alternative.
What's the opinion here? This question only pertains to OTP development.
It's just a matter of using the right tool for the right job. There's a lot of things Docker can do that an OTP application already handles (and in a way that's better-integrated with your application). Same with an orchestration system (e.g. Kubernetes).
You can use Docker to provide a common/consistent base on which you'd build the nodes running your OTP applications. If you have zero non-OTP dependencies, then this is overkill (you'd be better off deploying directly to some server, be it a VM or bare-metal, and managing configuration/deployment on that level), but most real-world applications do have other Dockerizable dependencies (databases come to mind, though there's ongoing debate on whether or not putting Postgres in a Docker container is actually smart). Also, using Docker to abstract the runtime environment for BEAM does help considerably with deployment to various PaaS platforms (e.g. Heroku, Bluemix, Elastic Beanstalk, etc.).
Personally, I lean toward a couple big servers dedicated to BEAM instances (running all the OTP apps), and a couple big servers dedicated to some SQL database (usually Postgres). Whether you maintain those servers as Docker containers or EC2 instances or DO droplets or physical servers is an implementation detail that doesn't really matter that much as long as it works and it's easy for you (or your DevOps / sysadmin teams) to maintain.
On that note: I don't think it makes sense to have a bunch of little containerized microservices each running a separate BEAM and a separate set of OTP applications. OTP already takes care of that sort of orchestration, and OTP applications already are (or can be) microservices. Just throw 'em all into a big server and let BEAM's scheduler and OTP's supervision model do their jobs.
If you get to choose your infrastructure that may be fine. If you don't and you work in even a medium size company you likely work with what they have, which is increasingly kubernetes.
That said, they still do not do similar things. OTP does not take care of orchestration if you must run more than 1 node.
You are still benefiting from OTP's supervision and scheduling when using docker or k8s, the difference is in what other features and the size you need.
No one is saying always deploy to k8s, but they are not overlapping.
Nor does using BEAM make it uniquely fitted to running on physical machines compared to other languages like java, go, C++, Rust, ... If your application is suited to running on a few droplets the language you are using is not likely the deciding factor for that.
> If you don't and you work in even a medium size company you likely work with what they have, which is increasingly kubernetes.
And that's totally fine, per my second and third paragraphs.
> OTP does not take care of orchestration if you must run more than 1 node.
No, but it (with BEAM) gets you most of the way there. The only thing not explicitly handled is spinning up the machine running that node (though you can certainly have the app itself drive that, e.g. by using the AWS API to spin up another EC2 instance with an AMI preconfigured to boot up and start ERTS). Once the machine is running, the application can use that node as a spawn() target and start throwing processes at it.
> Nor does using BEAM make it uniquely fitted to running on physical machines compared to other languages like java, go, C++, Rust
BEAM (or at least some sort of Erlang VM) is (in conjunction with an application framework that can capitalize on it, like OTP) is exactly what pushes the needle toward running on a couple big machines instead of a bunch of little machines (and again, whether those big machines are containers or VMs or physical servers is an implementation detail). Java and Go and C++ and Rust don't have Erlang's/BEAM's process model; they're dependent on the OS to provide concurrency and isolation (though there's no particular reason why a JVM implementation couldn't support an Erlang-like process model, on that note; similar deal with a .NET CLR implementation, or any other VM).
The compelling reason to go with a bunch of small machines instead of a couple big machines is resource isolation (e.g. enforcing memory/disk/CPU quotas for specific applications). It's definitely possible to do this on an OS process level (i.e. by applying those quotas to the BEAM processes), but if you're already using something like Kubernetes, no reason to not use that particular hammer to smack that particular nail.
Hi, one of the authors here, the Docker chapter and soon to come Kubernetes chapter hopefully shed some light on this very question. They are not alternatives, docker and k8s complement Erlang.
Do you have experience with performance optimization of Erlang code running in Docker? Is there a way to make the BEAM schedulers sticky? Does it make sense for latency reasons? I know that some products rely on core stickiness for maximum performance. Maybe Erlang/BEAM is not in that category.
There are some performance related topics covered in the Docker chapter regarding schedulers. And there are improvements already merged for Erlang 23 to remove the need for as much manual tweaking -- for example in 23 the VM will only start a scheduler per-cpu the cgroup actually allows, so if you are limited to 2 "cpus" by the cgroup there will only by 2 active schedulers.
Erlang does have the option to bind schedulers to processors but I don't know of any real world usage of it:
It’s difficult to deploy Erlang apps in Erlang-style using Docker. The style I speak of is zero downtime deploys where the VM is never turned off, but is updated with new software as a living organism.
You can run BEAM apps in a container the same way you’d run any other app. It’s possible. But BEAM is less like a PHP/Python/Ruby etc interpreter and more like an abstraction for the OS (some might even consider it to be one). When you’re in Erlang you think in Erlang and the runtime abstracts things like processes, threads, and network partitions away... so you tend to want a big beast BEAM instance (or a few big beasts) rather than a lot of tiny ones (as one might scale up a horizontally scaled app like one on Heroku dynos).
It can be done. It’s just kinda going against the grain.
It is not going against the grain. Using release upgrades (upgrading a release without restarting the VM -- unless the version of erts has changed) should only be used where necessary and they rarely are.
They are great to have when you need them but should not be chosen lightly.
Part of the point of this booksite is to show integrating Erlang into a modern environment and not being the oddball deploy at the company. It is the lack of libraries for services used for deployment/monitoring and lack of documentation that is the reason for this.
Also, there is no abstracting away network partitions.
I deploy the Erlang system I use at work using Docker and it's working very well for me. Like you say, it's not a zero-downtime relup but rather just replacing one app container by an updated one. It was a little bit tricky to get the image built properly (the right combination of dependencies, ERTS inclusion, etc) and things like remote_console are slightly more cumbersome than they would be otherwise (it becomes a docker exec command) but overall I feel like the advantages (mainly very smooth deployment and declarative configuration) outweigh the disadvantages.
This is what I've been thinking. At the very least, I know that Docker interferes with the BEAM VM's default behaviour for port management, and node discovery. Docker containers require developers to select which ports containers communicate through. Isn't that right?
This is ultimately not very different than what you would get with any firewall or network rules in any business, and it's an issue shared by most deployments regardless of whether you get containerized or not.
> I also tend to believe Docker makes little sense, where OTP releases are the alternative.
Docker makes a lot of sense, actually, because it allows you to run integration tests easily and ship Erlang releases in the same way as everything else. There are exceptions, of course. As for OTP releases, they are a good way intermediate step before packaging code to RPMs, but upgrades should be done the usual way: via blue-green deployment. Don't even argue with me :-) I happened to support a huge system that uses OTP upgrades, and the amount of complexity and tricky bugs, that live code loading introduces, is immense. No one is smart enough to do it :-)
Was going to say something similar - having a consistent, predictable deployment environment is really valuable from a support perspective, especially if you are shipping a service to several end users (as opposed to deploying an in-house service).
> If you expect failure to happen on an external service, do not make its presence a guarantee of your system. We’re dealing with the real world here, and failure of external dependencies is always an option.
What is the use case for erlang given the current container technology and cloud providers? I mean many problems that erlang solved (and beautifully did it) can be dealt with docker and kubernetes
In a nutshell, docker and k8s are to OTP what region failover is to k8s tech. They operate on different layers of abstraction and impact distinct components.
OTP allows handling partial failures WITHIN an instance, something K8s can’t help with.
I agree that some of the problems that OTP solves can now be taken over by Docker/K8s if you modify your architecture a bit. I.e. if you only have a few processes running on BEAM it's likely you could replace them by a few containers with a supervisor that restarts containers with errors, though you would have to start thinking about how they communicate (HTTP?) and some other things. It wouldn't scale to the same extent either (you don't want 1000s of containers on a single node) and Erlang provides some other benefits as well (e.g. single interface to manage all your processes, consistency if all your services are in Erlang, efficient threading). I would say that the main use case is when the model of using 1000s+ of processes helps you build an easy to understand architecture, which can be in messaging, streaming, gaming, or other applications with many simultaneous users/clients.
It's an ongoing effort to gather all the resources that will help you use Erlang in a business. The booksite is divided in three sections focusing particularly on Erlang/OTP’s higher level concepts in the current open source ecosystem, how to use it in production (while setting up a pipeline for continuous development and delivery), and how to build a team when you’re starting from scratch.
We're working mostly linearly in each section, but they're not all progressing at the same pace. We have nothing yet on "building a team" although our outlines are all planned; the "hard things to get right" chapter was supposed to come in later but we saw enough questions about these topics to rush it out.
We might release things from each section as they are ready and we see fit. In the meanwhile, we're always appreciating feedback and people checking in to see what's new in terms of content.