This is a tragedy of a product being developed for an audience of zero.
All of these problems have acceptable, established, and time-tested solutions using Node.js itself, without the crippling caveat of lock-in to a new and unproven platform. And yes, despite the copy trying to distract from this fact, this is very much a case of lock-in, because built-in multithreading is not a Node thing. None of the mountain of Node code out there works on that expectation. Furthermore, JX seems to have its own proprietary packaging solution, SQL database, memory store, monitoring, and more. This isn't a point in its favor; it's just more lock-in.
These new features might be great and well-implemented, but if I'm going to benefit from them, I'm leaving Node.js behind and trusting that JX will be supported as well as Node itself is. I have no faith that will be the case. It's (perhaps unfortunately) and self-fulfilling prophecy: I doubt I'm alone in my thinking, and that feeds back into lack of a community coming together to support this fork.
My advice to the developers (who are clearly talented if they can put together something like this) is to refocus your energies on building stuff with (and on top of) Node.js instead of forking it.
A lot of wisdom there. I had similar experiences with SocketCluster.
It used to be a full-stack framework (with a heavy, opinionated client-side part) but I was the only one working on it and when Meteor, SailsJS etc came along, I understood it didn't stand a chance so I pulled out the realtime and clustering features and used them to focus on just the realtime part and it picked up!
What I noticed is that the more opinionated (and more complete) your framework is, the harder it is to build a community around it.
Developers like to use custom combinations of small, SPECIALIZED tools that only handle a SMALL part of the big problem.
... Unless you're a HOT startup with crazy funding... In this case, developers will trust your product by default. Developer trust is hard to gain - If you're small, you have to play for the long term and keep reinventing and rewriting parts of your project over and over as technologies change.
I'm not seeing what this has to do with Docker, other than the planned future ability to run M/R jobs inside Docker containers. That's an interesting idea, but I don't think enforcing Docker has any advantages; if you have a clean and simple API like REST or even plain pipes, you open yourself up to a whole new world of composability without needing something relatively heavyweight like Docker.
Additionally, you seem to be leaning on CoreOS at the moment. That seems a dangerous dependency considering the CoreOS/Docker relationship.
> if you have a clean and simple API like REST or even plain pipes, you open yourself up to a whole new world of composability
Totally agree with this and that's one of the core tenants of our API design. We should probably be more clear about how docker fits with pfs. Our APIs are all designed as RESTful services to allow for composability, however we want to take a batteries included but removable approach. In our case Docker is a battery. We want it to be there so that users have a really easy primitive to implement M.R jobs with. But we recognize it might not be for every user so we also want to allow people to put anything they want there. I think the easiest way would be just letting people pass an arbitrary endpoint to be used in an M/R job.
> Additionally, you seem to be leaning on CoreOS at the moment. That seems a dangerous dependency considering the CoreOS/Docker relationship.
I'm hopeful that both of these companies commitment to a batteries included but removable approach will make leveraging both ecosystems a realistic option. I agree that it would be a pain to have to pick one.
Am I correct in assuming this means I can (eventually) use PFS without any dependency on Docker? In particular, I'm interested in knowing whether I can expect to be able to run PFS contained within an arbitrary unprivileged container and use my preferred orchestration around it. Or, is it the goal of PFS to take over the orchestration plane, or require CoreOS's? Or, something else?
> I think the easiest way would be just letting people pass an arbitrary endpoint to be used in an M/R job.
We very much don't want to take over the orchestration plane. We'd much rather interoperate nicely with existing orchestration systems. We just want to add the ability to store and access large datasets within these existing systems.
Your ideal of being able to use and arbitrary unprivileged container and preferred orchestration software is how I feel it should work eventually as well. Unfortunately right now we have to target very specific environments though so we can focus our development efforts so eventually may take a little while.
It's great to hear about these concerns early on so thanks for taking the time to comment. I'll definitely make sure that we hold on to this as a core tenet.
Actually I think what you're referencing is still Dunning-Kruger: "Conversely, highly skilled individuals tend to underestimate their relative competence, erroneously assuming that tasks which are easy for them are also easy for others." (from wiki)
More specifically, it's the Downing effect: "One of the main effects of illusory superiority in IQ is the Downing effect. This describes the tendency of people with a below average IQ to overestimate their IQ, and of people with an above average IQ to underestimate their IQ." (also from wiki)
I think it's a pretty good analogy. I thought it was well understood that Michael Jordan was a very competitive individual and didn't take his wins for granted and that there was always something to work towards and improve upon.
"...the most competitive individual...Always felt like somebody else was going to outwork him so he wanted to out work them." - A trainer of Jordan's. (Video for reference: http://youtu.be/w39-_rauFSk)
usually, as a shooting guard, jordan would be playing with 3 people taller than him on his own team, and only 1 person shorter than him.
just like basketball is not purely about body size and athleticism, chess is not purely about iq. And it is kind of idiotic that iq is the thing that people think must be the one thing that makes good chess players good.
after a certain point, iq probably doesn't matter. motivation, concentration, competitiveness, determination, experience, preparation, risk-taking, creativity, pleasure when thinking about chess... i really don't understand why iq is the thing that's talked about so much.
magnus clearly understands that some people will neglect chess for something else. The implication is that there are geniuses out there today doing other things instead of moving figurines over a grid... that's a very humble and wise world view.
I'm actually surprised virtualized page-zeroing isn't a memory controller feature at this point. With all of the things modern CPUs do or you, it seems crazy that everyone is running CPU threads that waste time and voltage to pump a zero page queue.
"The big gain for this is that the OS doesn't have to explicitly zero out new pages, which would be a lot of bandwidth and time, and accesses to uninitialized memory only take the time of the cache and TLB lookups instead of having to do memory round trips."
The Mill [1] does/will do that: When memory is allocated, it is served directly be the cache, which implicitly zeroes the respective cache line (without involving the main memory).
A similar mechanism zeroes a stackframe upon a function call (and function return, I think), which eliminates most attacks that exploit buffer-overflows.
So a Mill CPU can never do shared-memory process parallelization? Two or more processes accessing the same memory could be a hazard. Seems like it would suffer the same issue IA-64 has with parallelism. However, the conveyor belt analogy simplifies register spilling & related issues.
It's a little bit silly, but zeroing a page is an extremely cheap operation (far cheaper than just about anything you are reasonably going do with the page once it's zeroed -- on the order of a hundred cycles is pretty typical these days). That said, yes, it is a cost, and it's not crazy to want to address it with HW.
FWIW, both powerpc and arm64 have a "just zero the damn cacheline" instruction. That's not quite what you want, but it is quite useful.
A hundred cycles to zero a page? If 4 KB can be written in a hundred cycles then, assuming a 3 GHz processor, that's 30 million pages per second or ~120 GB/s. That's pretty fast memory. On x86/x64 processors, which lack a zero-the-cacheline instruction, the memory will also end up being read, so you need 240 GB/s to clear pages that quickly. This ignores the cost of TLB misses, which require further memory accesses.
The zeros don't need to get pushed to memory immediately. They go to cache, where they will typically be overwritten with your real data long before they are pushed out to memory. That push of your real data would have needed to happen anyway, so there (usually) minimal extra cost associated with the zeroing.
There are, of course, pathological cases where you touch one byte on a new page, and then don't write anything else before the whole thing gets pushed out, but they are relatively rare in performance-critical contexts.
The whole point of cache is that you don't need to go to main memory every time you do an operation. If you don't write anything else to the page, those zeros will be written out to main memory eventually, but if you weren't going to write anything, there was no need to allocate the page to start with. So the most common scenario is the zeros get written to L1, which is fast, and then your real data overwrites it in L1 before it's pushed out to lower levels of the cache or memory. The initial write may require a read-for-ownership from memory, depending on the exact cache semantics, but if that's required, it would be required for you to write your data anyway.
Times change, and technology changes value. Scored music "screwed" street performers, then recordings "screwed" concert performers, then the internet "screwed" recording artists.
Plenty of artists have successfully adapted and are doing great. Others have sheltered behind curators and promoters like labels. Others have failed to deliver what people want and they are screwed. That's capitalism at work.
I don't understand why we should hold back progress so that existing artists can be sheltered from needing to adapt and provide a product people want to pay for.
I say this as a former small-time producer of club tracks and a current software developer.
Additionally, you should ask artists who's screwing them before you start pointing fingers. Many artists would kill to have their music downloaded. Much of the screwing comes from the big labels that reap almost all of the profits.
> Plenty of artists have successfully adapted and are doing great.
Exactly. The artists that have adapted are the ones whose art is actually valuable, rather than artists who work with the people who control distribution channels. Distribution channels are a lot more prone to disruption than creators of art.
"I don't understand why we should hold back progress"...
I don't think we should hold back progress. I'm not arguing for legislated compulsory DRM or shutting down the Internet.
I'm just saying I don't have a lot of love or sympathy for people who build businesses on the appropriation of peoples' hard work against their will. I feel the same way about jerks who take OSS software closed without authorization or credit, or who scrape peoples' blog posts and use them for their own click bait sites without even linking to the original author. TPB is in that sort of category.
"Additionally, you should ask artists who's screwing them"...
I have, and trust me... nobody has any love for record labels except maybe some of the indies.
The trouble is that the new model -- the promised land -- is not appearing. There is no new model. The new model is you give your work away for free and starve.
Like I've said, we've gone from a system where there was some opportunity -- albeit in a shitty model with shitty record labels -- to a system with no opportunity outside touring and merchandise sales.
You can squeeze by on that, but you can't build a career on it. Great art requires years of dedication. That means making it a sustainable career, a vocation. Squeezing out a few tunes in between your two day jobs just won't lead to great art.
It also means you're basically a consultant -- a glorified service sector employee. Why is it that programmers are allowed to build equity in startups, but artists are forbidden from building equity in a portfolio of copyrighted works? It's the only vehicle for equity building they ever had. Without some opportunity to build on your work in a non-linear fashion, you are just a wage slave forever.
I guess we're going to reward all those artists and musicians who enriched our lives by making them eat dog food when they're old...?
Sonny Bono always gets bashed for saying copyright should be "forever and a day." I still don't agree with him-- I think that position is too extreme. But I understand where he's coming from now. He understands economics and business.
To build an industry that can pay actual wages, benefits, and invest in new things, you need to build capital. You need some way of building a portfolio of enduring value that can produce recurrent revenue over a long period of time to fund new things and to support the vast overhead that a real profession demands.
Nobody should understand this more than the HN crowd, which is why I have such an uncharitable interpretation of the down votes I always get for pointing these things out.
I agree with you that it's troubling that a new model for monetizing free work isn't coalescing -- in art, open source, writing, and more. I've personally felt the sting of all of these.
However, I don't blame the downloaders or the Pirate Bays. People want what they want, and Pirate Bay managed to provide that to people: a download index for the price of an ad or a drive-by download. I congratulate them on their success.
I think if you can't convince people to hand over money for what you've made, then what you've made simply isn't valuable, even if it maybe was yesterday. Perhaps this means that the local band will have to pack away their guitars while the one-hit wonder Gangnam Styles rake in the bucks, but nobody is being screwed here.
What society wants from entertainment has evolved, and holding onto a static definition of what constitutes "art", "quality", or "enriching one's life" while ignoring what real people are clearly demanding from their entertainment is, I think, pretentious.
I don't think the internet is destroying anything or screwing anyone. It's just closing gaps and optimizing every industry towards exactly what people want; if that's thrown-together garbage delivered for free at the cost of promotions shoved into your eyeballs, then don't blame the internet, or Pirate Bay. Blame people for consuming crap and blindly selling their souls to ads.
> I think if you can't convince people to hand over money for what you've made, then what you've made simply isn't valuable.
They aren't paying because there is pretty much a zero percent chance of getting in trouble for downloading it illegally.
If magically somehow tomorrow every illegal download came with a bill for 250$ for each infraction at the end of the month things would change and more people would move back to legal purchasing. People are cheap and if they can take something for free without consequence they are going to do so.
The myth is always that either one illegally downloads or pays for it through channels. However, most people I know will pay for some content and not others. It's a question of value. Just because the illegal download costs $250 does not imply that people will pay $10 or $20 instead. The third (and oft-ignored) option is that they will simply find other content that is worth the expense (both monetary and effort).
Who was making a living recording and selling albums before the internet? As far as I know, touring was always much more lucrative than recording. Labels always made much more money than artists and the artists were ok with that because the only way to get famous(and make money touring) was for a label to like you and promote you. Things have changed. It's harder than ever to make a living playing music because less people want to see live shows, but that isn't thepiratebay's fault. At the same time, it is easier then ever to get your music out there. If pirating was completely stopped it wouldn't change anything for "artists". We'd just revert to labels choosing winners, but it would still be harder to make a living now then it was 30 years ago.
> Like I've said, we've gone from a system where there was some opportunity -- albeit in a shitty model with shitty record labels -- to a system with no opportunity outside touring and merchandise sales.
> You can squeeze by on that, but you can't build a career on it.
Steve Albini strongly disagrees with your theory [1]:
"...As a result fans are more ardent for this music. They are willing to spend more on seeing it played live. They are willing to buy more ephemera and eager to establish a personal relationship to the people who make the music. Gig prices have escalated as a result. And the merchandise tables at gigs are universally teeming with activity. Back home, gigs that used to cost five or six bucks are now 20 or 30. Over here the ticket inflation has been more pronounced, with club gigs going for $80 or more. As a result gig income for bands has increased exponentially. My band has been playing a lot of the same places for the entirety of our existence, over 20 years now. I guess you could say we’ve saturated our audience, no matter how long we stay at it. Some of these perennial gigs are now paying an over of magnitude better than they were 10 or 15 years ago. That’s right, some places where we used to earn four or five hundred dollars we now earn four or five grand."
In an internet-ubiquitous world, the idea of restricting the flow of data -- any data -- is basically a magical fantasy of those who understand neither the technology nor its sociology. The Pirate Bay will be back in a day or two tops.
I wonder how much longer people will keep trying to put the information genie back in the bottle?
" is basically a magical fantasy of those who understand neither the technology nor its sociology."
Ever hear of steam, MMO's, F2P? They've started with backend chaining of software, they are going to try to turn everything into "cloud based" bullshit to as much as they can do it.
But I'm not going to grumble about it because I don't see this as a startup. In a social kickstartered world this stuff is the new breed of entertainment.
And the way you evaluate successful entertainment is buzz, not creating "business value" or whatever metric you deem more real.
His argument, if you read his work, isn't based solely on the increasing speed of computers.
His idea is that progress is exponential in all of the requisite areas. That includes algorithms, hardware, biology, neuroscience, and more.
> Siri, etc, are nothing more than text parsers that give a canned set of responses
We say the same about everything once we can do it with computers, because progress doesn't come magically. It's incremental. Yet much of what we have already is what used to be "science fiction" and, before that, "magic". I'm sure when AIs are passing the Turing test, we won't think any more of computers, but less of the Turing test.
This isn't an argument against Kurzweil's predictions, it's just moving the goalposts as you say. If strong AI comes, we'll move them all the way there and maybe even a bit past.
I disagree that we'll know it when we see it though. I think we'll deny it until we die and the new generation grows up in a world in which computers have rights.
> I'm not saying it won't happen, but it will require a type of conceptual breakthrough that we simply haven't had yet.
Finally, Kurzweil's arguments do not require a breakthrough as a premise. Kurzweil's idea is to scan the brain at the neuronal level, then brute-force its simulation at the biochemical level. It's straightforward extrapolation.
You could propose that we will come across some roadblock in our exponential progress towards that goal, but in the absence of one, the null hypothesis is that progress will continue as it has. Then indeed, the singularity is nigh.
All of these problems have acceptable, established, and time-tested solutions using Node.js itself, without the crippling caveat of lock-in to a new and unproven platform. And yes, despite the copy trying to distract from this fact, this is very much a case of lock-in, because built-in multithreading is not a Node thing. None of the mountain of Node code out there works on that expectation. Furthermore, JX seems to have its own proprietary packaging solution, SQL database, memory store, monitoring, and more. This isn't a point in its favor; it's just more lock-in.
These new features might be great and well-implemented, but if I'm going to benefit from them, I'm leaving Node.js behind and trusting that JX will be supported as well as Node itself is. I have no faith that will be the case. It's (perhaps unfortunately) and self-fulfilling prophecy: I doubt I'm alone in my thinking, and that feeds back into lack of a community coming together to support this fork.
My advice to the developers (who are clearly talented if they can put together something like this) is to refocus your energies on building stuff with (and on top of) Node.js instead of forking it.
Source: some hard-learned lessons