What specific features of VM/CMS are you thinking of when you say that?
If anything, I would say it is the other way around. The classic MVS filesystem is more hierarchical than the classic CMS filesystem (when I say "classic", I mean ignoring newer innovations such as SFS, BFS, HFS and zFS). The components of MVS dataset names resemble directories, even though they are mostly (but not entirely) just a naming convention. By contrast, the classic CMS filesystem (pre-SFS/BFS) has an essentially flat namespace on each minidisk – its file naming is closer to PC-DOS 1.0 than to Unix. So, even though both filesystems are quite remote from Unix, the MVS filesystem is a little bit closer to Unix's than the CMS filesystem is.
Also, if you compare the support for multitasking on MVS to VM/CMS, I think you'll find that MVS is closer to Unix. A single MVS operating system image has multiple address spaces (processes) each containing one or more tasks (threads). By contrast, VM/CMS works by running a single-tasking operating system (CMS) in separate virtual machines on top of a hypervisor (although more recent CMS versions have added "application multitasking" which is roughly equivalent to threads). Although neither is exactly the same as the Unix model of processes and threads, MVS is much closer to Unix than VM/CMS is in this area. Given that, it is actually much easier to implement the Unix process model on top of MVS than on top of CMS, which I suspect is why MVS OpenEdition / z/OS Unix System Services comes with a full-featured fork() implementation (which passes the Unix conformance test suite), while CMS OpenExtensions' fork() isn't really fork() at all, it is actually vfork() pretending to be fork().
I've not used CMS, though I have used, long ago, MVS.
My understanding is that CMS followed in its genesis and development spirit a bit more of the Unix development mode. It was more popular among (at least some) users, and as I understood it, a bit of an ugly stepchild / embarrassment to IBM, who considered MVS the more buttoned down, business-oriented, product.
This is based on some long-ago, hazy, second-hand accounts, mind.
I agree with your recollections in that VM/CMS had culturally more in common with Unix, even though technically they had very little in common at all. In the 1970s through 1980s, VM/CMS saw a lot of use as a research computing platform, there was a lot of scientific software available for it. VM/CMS was used as a development platform by many groups within IBM, when some other platform was the target–for example, developers writing the AS/400 operating system, OS/400, wrote their code and compiled it under VM/CMS and then transferred it to AS/400 for execution/testing. (Later they moved to AIX instead; OS/400 is somewhat unusual in being an operating system that has never been self-hosting, never been able to compile itself.)
VM/CMS tended to get new features added to it organically, because some user or team felt the need and just implemented it; the product direction of MVS was much more tightly controlled by IBM management. So VM/CMS was the birthplace of IBM's mainframe email products, peer-to-peer networking (used for the BITNET academic network), the REXX programming language, pipelines (inspired by Unix pipelines but different), etc. Many of those features were later ported to MVS.
In the early 1980s, IBM planned to ship workstations running VM/CMS, hoping that academic/scientific users and developers used to VM/CMS on a central mainframe would like to run it on their desks. This plan never really took off, in part because a lot of those users were already moving to Unix, and Unix workstations had a better price-performance ratio; if Unix wasn't around, maybe IBM's plan would have been more successful.
Thanks, yes, that corresponds to my recollection of the story.
Unix in many ways is the result of its development model, and the "scratch an itch" nature of it, by people capable of scratching those itches. VM/CMS seemed to follow in that same tradition, and as such, addressed (an admittedly technical) end-user community needs in ways that a "delivered" system, in the sense of the providers and users being generally distinct communities, doesn't.
I don't know that VM/CMS would have filled that niche, but it seems that it might have. Having used VMS and Unix, the former, though often compared to Unix, strikes me as far more a "delivered" product. There was a time in my life I could spell DCL (also JCL), and even write some basic utilities in both. That's long since passed.
The capabilities of a Unix shell are far more friendly to ad hoc development and needs -- I'll gin up new scripts or shell functions all the time in ways that just didn't flow from the other languages.
(Now, if I'd picked up, say, REXX, I might feel differently, though I'm really not sure.)
