I use Graphviz dot syntax to configure entire cloud computing environments in the order they must be built. This is pipelines as code and infrastructure as code
It's very powerful! Not every organisation can bring up an entire environment with one command of documentation their infrastructure with Graphviz.
Here is a executable diagram of a build worker cloud nodes, Kubernetes, consul, vault, Debian package server, Java app, SSH bastion, Prometheus, grafana, DNS and security groups.
I am big on state machines. Even simple tooling around them makes life easier for a lot of tasks. We often poorly implement a state machine in our code. Formalizing it in some way often makes life easier for me.
I created an analysis compiler that takes intent information and makes it part of the Enterprise DevSecOps pipeline in order to share realtime analysis with the entire organization. As I continued that work, I've also started talking about the feasibility of creating a supercompiler.
GraphViz is the shit, man. It was exactly what I needed to work on this material. You can create your own notation and then couple it with all kinds of tech by way of batch. Highly recommended.*
*The hardest part, frankly, was unlearning all of the other diagramming paradigms I've used. YMMV, but I found GraphViz most useful when you're going completely greenfield.
Application performance monitoring, scheduling, control loops ala Kubernetes, parallelism, multithreading, data scheduling (shifting and sharing data to where it needs to be used), advanced caching and regeneration and container scheduling and binpacking and data layout scheduling could all be compiled by a supercompiler for extreme performance.
Our industry tends to over-complicate things and get in its own way of making the world a better place. SuperCompilers are a worthwhile effort to fix that. We need more forces for good. Kudos.
1. Whenever anyone in the organization has an interesting conversation about the product they take notes. They're just plaintext notes using notepad or whatever.
2. There are about 20 keywords. They are represented as ALL CAPS
3. When you take notes, you tag them with the keywords
4. These notes are checked into git like everything else, and built as part of the dev/team/pmo/enterprise DevSecOps pipeline
5. Doing this creates a realtime and ever-changing analysis model
This leads to better conversations, better questions, elimination of duplication and conflicting business ideas, entry of data once, tracking bugs and features in plain language all of the stakeholders can understand, automated testing frameworks, interactive diagrams showing what the business thinks the world looks like and how they're products will change it, and more.
It was fun trying to design the simplest system possible able to map the almost endless complexity of human interactions around a given domain. tl;dr "Intent" is everything that happens between humans to organize their view of the world and the tech they're making to interact with it. Intent begins with brainstorming, perhaps BS sessions with coders, marketers, customers, and so forth and ends with feature tests, user manuals, and everything but the code itself.
I’ve kind of considered something similar to this approach but ended up using a graphdb to keep dependencies instead.
Using a db gives you a better way of modifying meta data in batches which might be required at scale.
It also makes it possible for tooling to query stuff quickly and do operations such as setting consul intentions, register something in an api gateway or other surrounding config tasks.
The pipelines we built checked out complete subgraphs from an api and generated json that referenced versioned TF modules and committed these to a service repo.
Thought about open-sourcing for a long time but never could get the go from legal. :(
Holy macaroni, this is one of the coolest things I've seen in a while. I wonder if a top-level post of this could generate some interesting discussions about other people's exotic approaches to "DevOps", too?
Unfortunately I tried promoting my things too much so most of my submissions are blocked by the self promotion filter.
Yes I would love to hear other people's solutions to DevOps problems. I find it an interesting space. There's lots of good ideas but they haven't been brought together.
o wow! i know that graphana can be used with mermaid diagrams, but this is a whole new (lower) level. it would be nice if the graphviz could be styled a bit nices (like with sketchviz), but that's icing on the cake.
I love GraphViz. Many are saying the layout isn't that great and there are better options, etc. etc. and I have no doubt that for some use cases, that might be true. However, GraphViz "just works" for me. Every time I build a tree-like structure, I simply open up a new .gv file and just start typing nodes connected with an arrow ("->"). I can do this without taking ANY extra time - I just type it in as I'm developing the structure. When I'm done, I have a very nice diagram. For the more complex ones I add some color or other small details, but often they are simple black and white unidirectional graphs. I keep them open while I work and they help me visually see my structures which helps me see the problem domain more accurately and write better code.
.gz is already used by gzip[1]. This wiki[2] says it is .gv . I've always used .dot so I don't know what to use anymore.
edit: according to this[3] discussion , Microsoft uses .dot. I then found this wiki[4] page that supports the claim. dot is used by Microsoft Word document template files. I will switch to .gv
I've used .dot in the past and stumbled upon issues with git related tools (I believe git diff) interpreting the .dot file as a word related file. I'm using .gv now.
Great tool and nice to see such active development on it!
From 3.0.0 to 7.0.0 just during this year, though.. ? I guess very few are depending on the library interfaces which I imagine are being broken, or do the major version increments just signify nothing of importance?-)
It seems to me that they bumped the major version for every breaking change in the changelog. I guess that is in strict compliance with semantic versioning but indeed it looks strange.
You don't need to do a new release for every checkin. You can make a series of breaking changes and only then declare a new release, bumping the version number.
After all, it took (at least) 21 years to get from 0 to 3.0.0
There's absolutely no reason for 2 major bumps 11 days apart.
> You can make a series of breaking changes and only then declare a new release
That means you need to postpone the breaking changes for a while, kind of bunching them together. Many projects do that, but it also incurs some cost.
> After all, it took (at least) 21 years to get from 0 to 3.0.0
It's possible that they did not implement semantic changes in the strict sense before and included small breaking changes into non-major releases (I think a lot of software does this). Fast changes of version numbers may just mean they started reflecting each small BC break into the version number.
If you're doing CI/CD and pulling from head rather than official published versions, you deserve what you get. If you mean the graphviz project itself, version numbers are meaningless to it internally.
The thing is, if you're writing software with a lot of breaking changes, SemVer doesn't make much sense, because you'll just quickly get to major version numbers in the hundreds.
> The thing is, if you're writing software with a lot of breaking changes, SemVer doesn't make much sense, because you'll just quickly get to major version numbers in the hundreds
“The version number conveys whether there is a breaking change” seems more important than “the version number remains small”, so while the second half may be true, I don’t see how that supports the first half at all.
The problem isn't that it's small, the problem is that if few changes are breaking, then you have "version 5" which is significantly different than "version 4".
Is version 465 very different from version 464 though?
A bit unfair, no? Firefox switched to the "new major version every 6 weeks (or so)" only because Google did that first with Chrome, and people started thinking "Chrome is so much better because it's in version 17 and Firefox is version 3".
I liked GraphViz a lot, but eventually got frustrated with its layout algorithms and default look and feel. I would switch to Mermaid wholesale if it could render SVGs without requiring a browser engine.
I’ve personally settled on grapviz (a derivative actually, plantuml) being “good enough” for casual diagrams. Plantuml is built with the idea that it’s for quick and dirty communication, and I think graphviz translates well to this. It’s a nice way to put together a sequence diagram that just helps you understand control flow (and then never took at it again).
I have to admit, I haven’t found a more formal programmatic tool. Ultimately I have to turn to “by hand” tools if I want something more complicated. If you can’t translate that last statement, it means I just never do anything more complicated.
As I said before, I'm a big fan. I do wish there was a sister project that was intended for a higher level of detail. PlantUML has explicitly stated in some forum responses that it intentionally limits your ability to place elements exactly where you want them. I totally understand their decision, and I greatly enjoy using it for its intended use cases. There just isn't really anything that comes close to the practicality of PlantUML for more detailed diagrams, and I feel like PlantUML would be the best starting place for such a tool.
The creator of PlantUML is working on translating graphviz into a native form to eliminate the dependency for PlantUML use. Once this has been done, it would open up even a whole new world of possibilities (graphviz does allow a bit more specification of element locations).
Not sure if this question is directed at me. My understanding is that mermaid is relatively similar to plantuml in that it doesn't give you 100% control over layout and diagram features.
What plantuml lacks (maybe mermaid less so) could really only be improved upon by working with raw svg I think. There is a LaTex package tikz that offer much more flexibility than plantuml/mermaid, but you still run into limitations in that diagram features are predefined.
Ah thanks for your answer comfypotato, Tikz looks even better then Graphviz.
No, I misplaced my remark in the thread and directed at rcarmo just one level above who would have switched to mermaid if it didn't need a browser (or perhaps just node.js). Tikz also needs a LaTex dependency, with gigabytes of diskpace I'm personally not that frugal anymore with tooling size even though I like using as little disk space as possible.
Perhaps rcarmo reads the thread further, perhaps not.
Sketchviz looks a bit nicer, but also 'sketchy' https://sketchviz.com/new
Regarding mermaid, see my misplaced comment one level deeper. I'd think that with node.js it can export to svg.
Graphviz is excellent. I use it for (amongst other things) generating diagrams of states and transitions, and also for automatically mapping choices as I write interactive fiction.
More generally I've added it to Newt [1], which is my in-progress C# code generator. It scans a PostgreSQL database to automatically generate a class library with models and EF Core data contexts.
As part of that it writes out a dot notation Graphviz source file for showing the tables and relationships. The repo includes an example image [2] - it only shows a couple of tables, but it gives an idea of what can be produced (and also shows the .dot source file that was created to produce it).
Edit: If anyone does C and has skills or money available (I don't) the Graphviz README says "This project has only 3 of its founders remaining as volunteer maintainers, and support is limited and may end soon. If you can help fund to a more formal foundation to continue this project, please contact north on graphviz org."
You can’t mention commercial options without mentioning yWorks. Their product is absolutely amazing and I’d use it for everything if it wasn’t for the steep cost.
(Big fan of graphviz, it was such a shame when AT&T pulled funding!)
Maybe helpful, more for visual graph data analysis than say arch diagramming: we build Graphistry specifically for interactively analyzing more data than these tools, eg, GPU client + GPU server, and being more batteries-included. The free Graphistry Hub site is similar to diagrams.net and GitHub for folks who aren't self-hosting GPUs.
We are still working towards OSS'ing the core renderer, but all our new graph neural networks + graph compute + graph layout layers are all already OSS-first (see PyGraphistry), and we are part of the apache arrow + rapids communities. If your kind of thing, we are very much hiring :)
As someone working at yWorks on yFiles, thanks for the kind words :-)
Having used Graphviz extensively over the years as well, the cost of yFiles is certainly unfortunate, but I guess we simply serve a different markt segment than Graphviz or D3. (On a somewhat related note: Our online graph editor, yEd Live, support parsing a subset of Graphviz' syntax and that enables running at least some yFiles layout algorithms on Graphviz graphs. No export back, though.)
I really struggle with this. I also use networkx, Gephi, and graphviz, and occasionally plotly. D3 is too much of a learning overhead for me at the moment.
I'm really in the market for a decent engine that's both easy to use and enables high quality interactive exploration of massive datasets. Plotly comes close, but struggles with larger datasets and is still a bit fiddly. I'd love to be able to lob a large structure into a UI and share with colleagues, and they'd be able to visualise and filter based on node/edge metadata, time series etc.
Something like Gephi, but web-based and considerably easier to use.
Agree. Neo4j’s visualization tools are surprisingly painful to use. I’d love to find something that allowed you to trivially add code to the layout engine. Graphviz’s clustering capability seems to be unique among any of the tools I’ve used as well.
Gephi can cluster, but the algorithm isn't that transparent or flexible. I've increasingly found myself doing the clustering in Python and rendering in Gephi.
Can be used code-free (drop in a csv or log file) or via analysis workflows like from Jupyter notebooks and internal dashboards.
We are roughly parity for Gephi's features, and pushing (far) ahead on web, GPU, perf, ai, automation, etc. Always curious about gaps - our Slack is a good place to chat on that.
Yes Omnigraffle accepted Graphviz files. But uses its own excellent layout engine and allows you to adjust the layout by hand.
If you include integration with OmniOutliner, you also get Graffle can also use a wide variety of text based outline file formats.
I bought it 5 years ago and haven't given them a dime since, and got a lot of new feature since (scripting!). I'd love to have more library, but that's another issue :)
There is an offline version: https://www.diagrams.net (they had to change their name from draw.io recently). There is a download button right on the main page.
Which looks to be an Electron app, so there's a browser engine doing the rendering. Can I save a diagrams.net/drawio-desktop drawing into a file, and convert that file into an SVG in a lightweight batch job?
I use graphviz via networkx to build a dependency tree for my company's app. I'm able to split one master tree into sub-trees based on which "flow" I want to look at.
Out of curiosity, has anyone settled on a method for storing and surfacing metadata about a node?
Example: a server has an IP address, dns entries, a description, and some other fields, and it connects to other servers. Is all of that meta information in a JSON object in the key-value area of the node in the .dot file?
Does anyone know of any DOT -> mermaidjs format converters? I was looking for one recently and was surprised not to find anything more robust than someone's quick search and replace script.
Does anyone want to look at porting the mermaid layout method back to Graphviz? It might be useful to people. I guess some friction would be around the kind of awkward way that Graphviz has to deal with objects that aren't strictly nodes, edges, or subgraphs.
Graphviz is one of those things I wish I had found much much earlier in my career. It makes throwing together a little one off tools to chart data relationships effortless.
"This project has only 3 of its founders remaining as volunteer maintainers, and support is limited and may end soon. If you can help fund to a more formal foundation to continue this project, please contact north on graphviz org."
The community should applaud the efforts of Magnus Jacobsson, Mark Hansen, Matthew Fernandez, Steve Roush, and recently Costa Shulyupin to support graphviz in a more disciplined and professional way. In hindsight, we got a lot of play by treating Graphviz as a research vehicle, but we weren't able to (or didn't choose to) direct a lot of effort into documentation, rewriting existing code, sorting out feature interaction, developing very polished front ends, or broadening the scope of the effort. Or finding a new home after AT&T Labs lost its enthusiasm.
I am glad to see that they recently (relative to project age) got two new developers which have for the last two years actively worked on rejuvenating the project, tackling some long standing issues, and actually making new releases with bugfixes. That message was added to README before that. Of course it doesn't mean people should stop supporting the project, but I am currently less worried about future of graphviz than the last time I looked into the state of it.
It's very powerful! Not every organisation can bring up an entire environment with one command of documentation their infrastructure with Graphviz.
Here is a executable diagram of a build worker cloud nodes, Kubernetes, consul, vault, Debian package server, Java app, SSH bastion, Prometheus, grafana, DNS and security groups.
https://github.com/samsquire/mazzle/blob/master/docs/archite...
The tool also parallelises the build based on the graph so for example packer builds can run in parallel.
https://devops-pipeline.com/
It also has a GUI but as a tool it's not ready for other people to use.