Personally, I've always thought Smalltalk's problem was the exact opposite: it tried to be a portion of the OS but not quite enough. I'd like to see a Smalltalk that took some tips from Erlang.
smalltalk was originally the entire stack, from the hardware all the way up, i agree that for a long time, it lived in a weird in between world where it was sort of an OS and sort of not. working out how to keep what makes 'smalltalk smalltalk' while ironing out those issues is something that both redline and pharo are trying to solve from different angles.
i'm interested in hearing more about the taking tips from erlang idea. i'd love to chat more via email:
> it tried to be a portion of the OS but not quite enough
That's actually a fairly recent problem (since the 90s maybe?) as Smalltalk implementors had to move onto "third-party" OS. The original smalltalk controlled everything but the hardware, much the same way Lisp Machines worked (yes I'm aware most lisps didn't work that way)