I've reached over 100 ideas too at https://github.com/cretz/software-ideas. Anyone can steal them for any reason (ideas alone have little value to me). Granted some of them may be more targeted towards certain technologies and less vague than OP's.
I’m somewhat struggling through getting a service working on Kubernetes. So much configuration to go through. I’m trying to build a product, not be a systems engineer.
It occurred to me that Kubernetes might really benefit from some kind of nodal editor. Think iOS Interface Builder but nodal.
It would let us visually see the network configurations, make connecting different bits easier, etc.
The editor would also show a visual representation of our current clusters, nodes, pods etc along with health and usage etc.
Man if that existed right now I’d pay some cash to use it.
I'm smiling at your comment, and please don't interpret this as me making fun of you; I'm laughing along with you at the absurdity.
> I’m somewhat struggling through getting a service working on Kubernetes. So much configuration to go through. I’m trying to build a product, not be a systems engineer.
I'm a systems engineer through-and-through. I've resisted k8s for a long time, but recently inherited a project that was already very k8s-native (for example, it's an Elixir-based app that auto-joins other OTP nodes based on k8s discovery) and figured it was time to dive in.
From my understanding, the whole point of this is to abstract away a lot of the pieces that were fundamental to "sysadmin" skills. I'm running this on a DigitalOcean Managed K8s cluster, so I haven't had to do any configuration on the nodes at all, but there's a ton of things that feel dramatically more complicated, even though they're nominally more "abstract" than they would be on bare metal. What's missing, though, are the systems power tools that help you debug what's happening! I have no doubt that this is an exceptionally powerful platform, but... when I start getting errors about a process not being able to make a TCP connection to a publicly-accessible IPv4 address, I can't just connect and look at the iptables configuration on the node to see if there's any kind of rule that might be blocking it.
In a lot of ways it feels like we're throwing out the baby with the bath water.
Yeah I’m setting this up on Linode’s beta k8s product.
Funny thing is that I know how to do mid-level server admin. Most of that comes from building and maintaining the servers for https://jsonip.com for years. iptables, nginx configs, etc. Building up a new Ubuntu server from scratch isn’t second nature, but I have a run book and enough familiarity that it isn’t hard.
I’ve even worked with Kubernetes a little over the last few years at a couple jobs. Dockerize my app, use some kubectl commands to get it deployed. Luckily I wasn’t responsible for maintaining the infrastructure.
Last night I had this realization. The last company I worked at is now publicly traded and has hundreds of millions of users. When I left they were just starting to experiment with k8s but we weren’t using it in production anywhere. There was an entire devops team working on it for at least a year and it still wasn’t ready for production.
So I realized that I don’t actually need Kubernetes. I’m super small scale and working on this project on my own. They scaled up to be a billion dollar company without it. But then it leaves me in the position of either hacking together my own limited auto scaling infrastructure or struggling through with Kubernetes.
I’m not really sure which direction to move tonight. I have enough personal funds for maybe 6 months to a year and this is just eating up time.
Do I struggle through a few more days trying to get this working with Kubernetes and “do it the right way” or give up and hack together a nodejs app that monitors system performance and auto scales for me as needed?
At least with the “hacky” app I’ll understand at a base level what’s going on. It’ll take the same time to implement, but when problems eventually crop up I’ll know what’s going on and how to fix it. But with Kubernetes I’ll forget most of the details after it’s working and fixing the inevitable issues will mean re-educating myself during a downtime. Neither is great, but it makes me lean towards DIY. And DIY is what systems like Kubernetes were built to purportedly replace.
There’s an expression at Amazon, adopted from our Indian colleagues: do the needful.
It means do what’s necessary to accomplish your objective and don’t worry so much about what the “right” approach is — that you should feel empowered to solve problems, not get caught up in the never ending stream of people yelling about what is “correct”.
Your goal is to release a product, and you have a tight deadline, so use the tools that let you accomplish that — what you’re familiar with, what you understand, and what enables you to work quickly.
If Kubernetes authors have opinions about how you work, they should show some customer obsession and release a product you can effectively use. That’s not your problem.
Your problem is releasing your app. Do the needful.
This explains a lot of the strange engineering I've seen from Amazon over the past several years.
Like why there's almost no coherence among the AWS services, or why the error-handling and documentation is never up to par, or why they're happy to throw away a library and replace it with another one with a completely different interface. Amazon doesn't need these things!
I will say that this behavior has generated absolutely zero loyalty in me. The only reason I use any Amazon service today is because I made that decision in the past, and it's more work to change. They look to me like the Blackberry of web services right now. It works, and they have millions of customers, but they simply don't comprehend how much better the experience could be.
I've re-setup my baremetal server I rent at hetzner in December to have a modern ops infrastructure. I started with k8s because it seemed pretty straight forward to have it up using rancher. After about a week of not being able to configure a simple hello world nginx docker image I gave up and tried using docker swarm.
I couldn't be happier with the decision: in less than a month of working on it during evenings and weekends I was able to set up a reverse proxy with traefik, a self hosted gitlab instance, a docker private registry and a full CI/CD pipeline.
With docker swarm, if you can 'docker-compose up ...' an app, you can also 'docker swarm deploy' it, with almost no modifications needed. Also if your needs grow and you really need k8s, you can 'kompose up' (or something similar, since I haven't got there yet) and have it deployed to a k8s cluster. But I imagine by the time my apps will need to run on k8s, they'll make enough money I will afford a devops team to handle the migration.
Feel free to contact me (info on my profile) if you think I can help you in any way having it set up.
> At least with the “hacky” app I’ll understand at a base level what’s going on. It’ll take the same time to implement, but when problems eventually crop up I’ll know what’s going on and how to fix it.
Being able to understand and remember a system is what allows you to control and operate a system. If you build a simple system, you may need to intervene more often, but most interventions will be simple. If you build a complex system well, manual interventions will be less frequent, but they are likely to be a lot harder.
> feel dramatically more complicated, even though they're nominally more "abstract" than they would be on bare metal
One problem I have with most of these systems is that they use completely foreign concepts. A programmer needs basic old-school sysadmin skills anyway just to develop a program. But most of those skills, however, aren't useful when using Docker on a server. You need to speak a very different language.
It reminds me of developing for small embedded systems. You get an IDE on your regular computer to write the program, but then after you've compiled it, you have to load it onto the embedded system and run and debug it there. Suddenly, the tools are all different, and most of them are crap.
k8s is interesting because its an enabling technology that also wraps tons of other complexity under its name. The execution model is straightforward enough but the networking model is complex for anyone not already familiar with ipvsadm, iptables, routing, layer 2 systems, overlay networks, proxies, dns, etc. Getting it running is actually easy enough to lull people into a sense of accomplishment. But if something goes wrong most teams wont know how to respond other than by restarting something.
I really hope we can agree on a simpler abstraction where we trade ease of use for less flexibility.
Kubernetes was not intended to be managed by humans.
The missing piece here is a set of operators that will automatically manage networking, storage. Those operators probably exist in GCP but were not open-sourced.
Spent the last couple days going through one blog after another and pages of k8s docs where the directions seem to fall to “copy this command that downloads a script from github and then follow the next step”.
I just want to improve my service’s reliability by adding auto scaling (Linode is my host) and k8s seems the way to go. Wasn’t expecting it to be so complicated though.
36. Documentation Linked Code sounds like literate programming [1], which interleaves the source code with prose explanations. This idealized description is due to Peter J. Denning [2]:
> A literate program contains not only the needed statements in a programming language, but also a precise problem statement, a summary of the background needed to understand the solution, an evaluation of alternatives, assessments of trade-offs between the running time and space, or between running time and programming time, and suggestions on how to modify the program. Program code segments are inserted in the text at points logical to the intellectual development of the algorithm. A literate program pays careful attention to lucidity of presentation and presents all arguments needed to understand why the program will actually run as intended.
I think I'm one of the few people who actually developed a Lotus Notes application. It was well received at the time doing standard (template) letters!
I think it’s awesome that you’ve created this list. I have thought about documenting my own ideas for years and never have because I’ve Never felt comfortable with it. It’s not that I think they are that great or that I’m afraid someone will steal them —- Id actually love it if someone did steal one or two. In any case kudos for putting your mind out there for all to see.
I was just adding some on-hover functionality to my side-project's website recently (to zoom screenshots, https://kopi.cloud) and was thinking I was going to need some "solution" for touch devices.
But then I tried it out on iphone/surface-pro and most browsers seem to work fine - you just have to touch the image instead of hover to get the animation to fire. Is there an issue with this I need to look into?
One solution is to use another touch screen (not showing pixels, just a touch pad) on the back of the phone. The front touch screen is for touch interactions, the back pad is for hovering/other interactions. It can be quite intuitive, and it is easy to use with one hand.
For the first one, about email, I have a few comments:
- For newsletters and forum/blog posts), I think NNTP can help.
- For invoices, a multipart message might help. This might also be used for having some format for delivery info, and for purchases.
- For replies to customer service tickets, you could use a part of the message ID.
For "Elements Represent Themselves", note that different users may use a different desktop environment and some none at all (I do not use a desktop environment on my computer). X selections could be used. For example, if the user clicks on a filename in another program, it could take the FILE_MANAGEMENT selection; if a file manager is running, it will notice that it has lost ownership of that selection and can use that to trigger a menu or something. In many cases, even the PRIMARY selection is good enough; the user selects the text and then enters some global key command which causes it to execute another program to do something with the selected text, whether it is converting a unit of measurement, ping a IP address, etc.
You could also have a similar idea for HTML documents; this is another use of the <widget> element I have described before that I think would be a good idea. For example, you could use <widget> to mark some text as representing a measurement. Since <widget> normally acts like <span>, the text is visible and you can still use document JavaScripts to provide this functionality too (just like any other <span>), although if the user has their own implementation, then it might provide more units to convert to (even if document scripts are disabled). And if neither <widget> nor scripts are implemented, it still works as static text!
Standard format for defining keybindings seems like it could work, although some programs may need their own extensions to the format due to various things, but even then, many common commands will be able to be copied without a problem, so it may still be helpful anyways.
I agree about 66 "Wizards and Forms Request Data". Furthermore, it should be possible to write the file locally and send it using a command-line interface, or even automatically convert data entered together with an uploaded file, for this purpose. (A scriptable interface makes this possible to do.)
For "Right Click Outsource", that may be good for a desktop environment; for people such as myself who use command-line interface, in some cases it can work. You could have one command which outsources audio, one which outsources directories, etc. (You could also have a FUSE to represent these things and then use a single interface with all of them, including a GUI if you want to.)
Mounting email attachments seems a good idea. A program could be made which allows mounting a mbox file in this way.
For 90 "Views of Data", SQLite has a virtual table mechanism, and also supports in-memory databases. These features could be used to do some of this stuff (and SQLite includes an extension to access JSON data in this way, actually; the command-line interface also supports importing CSV, too). But sometimes I want to be able to access remote data in this way and (and even make a join query between multiple remote data sources, such as movie times and weather, or any other combination that you might want). However, there are some deficiencies in the virtual table mechanism, such as the inability to have indexes on expressions (including nondeterministic expressions, which in the case of virtual tables, is sometimes useful), partial indexes, consuming LIMIT/OFFSET clauses, etc. (If it supports these things, the client could then understand such a thing as "ORDER BY RANDOM() LIMIT 1" and have it automatically convert into the request to access a random record.)
My side-project (https://kopi.cloud) can help with a lot of items from the article's list of stuff for email.
It started life as another "email burner service" - but for a weekend hack, I implemented "Email to RSS" functionality, and now it's a must-have piece of functionality for me.
Here's my process:
- hand out a unique Kopi address to a services
- block the address from delivering to my inbox and tell Kopi to publish the emails as RSS
- subscribe to the feed with an RSS reader (my preference is https://inoreader.com/)
I do this for a ton of stuff (linked-in, stack-overflow, newletters, etc.)
> X selections could be used. For example, if the user clicks on a filename in another program, it could take the FILE_MANAGEMENT selection;
I tried to bring (part of) this feature from Mac OS X (as it was called back then) to Linux / X11, back in the day. The problems are entirely social, not technical. You would not believe how strenuously people in the X11 camp object to crossing the application / window manager boundary.
The way I suggested would enable it to do nothing if it is not implemented (the application could check if the FILE_MANAGEMENT atom is defined, and if not, don't bother defining it since it is worthless; if it is defined and it takes that selection, but the file manager isn't running, nothing will happen, so that is OK too), so I don't know what is the problem with it.
But if the program supports highlighting text with the mouse and putting that in the PRIMARY selection, that could work too (and you can ignore the FILE_MANAGEMENT selection); the user might then define some global keyboard command or icon in the window manager or desktop environment, which causes it to display a menu of options for dealing with the file named in the PRIMARY selection. But in the case of filenames, the problem is determining what directory it is relative to. In the case of other stuff such as measurement numbers that you want to convert, using the PRIMARY selection in this way seem it would work; the application program need not be aware of such thing in order to enable it to work, since it is a separate program that would deal with the measurement conversion.
Either way, for people who do not want it, nothing happens! (I probably don't need it myself, but maybe some users use it, I don't know)
I’m on mobile and when my finger touches the screen, the page zooms out, and when I let go, it zooms back in—but off center. Anyone else experiencing this?
Edit: It seems this can be fixed by manually zooming out to 85%, but I still don’t know why this is an issue in the first place
