Norway took a several centuries to get back to ~1300 levels after the Black Death going from ~37k farms in 1300 to still only 16k farms by 1520, nation wide (and we're not talking large farms here, in most cases). Entire regions were entirely abandoned by the government. Many farms that were abandoned after the plague where not cleared until the 1800's.
If the population density was low to start with, and food production tough, recovery from even a single major enough event can take a very long time.
You blame it all on the black death, however, Norway was on the front line of the little ice age, so those farms might have not been populated because the land itself was not able to sustain cultivation reliably. Tolkien mentions only one bad winter, but I can allow that he wasn't well versed in climate sciences and regional climate patterns. As a result we have something that can suppress population growth to a minimal level, however, we don't have good proves in the text for such events in Middle Earth.
There's as far as I know nothing to support that the little ice age had much influence on the rebound of the population after the Black Death in Norway's case - you'll find documentation of farms right next to each other where one was not cleared and the other was, and where we can tell from tax records etc. once they were finally populated again that production capacity was comparable. It took centuries before the population had rebounded to the point where lesser farms were cleared again. As such it was first towards the end of the 1600's that the population was large enough to start hitting limits where the little ice age caused famine in Norway.
If the overall agricultural output is suppressed by LIA, then population recovery from the plague will be slowed down. From what I've found, most of Europe recovered population size before 1500. There is a Wikipedia article about medieval demography, but the data there is even more positive and I'm not linking it here.
My point was that overall agricultural output in Norway per capita did not constrain growth again until the 1600's. The Black Death is often in Norway in fact considered to have lifted a lot of the surviving poor people out of poverty, because people previously subsistence farming on tiny plots could suddenly take on far more or more productive land and produce an excess. As I pointed out, it was not until the late 1600's that Norway faced famines again. The Black Death significantly increased food safety for the survivors in Norway. Despite this it took that long to recover the population size.
> From what I've found, most of Europe recovered population size before 1500.
Hence the reason I pointed out that Norway did not, to show that the range of possible outcomes is substantial.
Differences for Norway included a harsher climate, and a landscape that in most of the country is not conducive to large scale farming in a constrained area (outside of a tiny part of the South East, even the "flat" parts are not particularly flat, and often rocky terrain), combined with a very low population density by European standards to start with. As a result, in a lot of places, while people had productive farms, the farm locations depends heavily on the terrain and are often spaced out, and a lot of focus was on e.g. raising sheep (which could, and still are, be let out to roam the mountain sides during the summer months). Recovery faced a lot of challenges, not least of which was simply that in parts of the country the next farm over might be many hours walk over tough terrain away, and during months of the year trying to go to the nearest village might be hazardous (winter months), or people might be up in the mountains far from people. There are still a few people living in places not reachable by road in Norway (and not just by the coast where they could compensate with access by boat). There are still large parts of the country where if it wasn't for cars getting from village to village might many places take days of walking (often horses or carts would not have been an option due to terrain).
Families could go the whole year without meeting any outsiders, and as a consequence marrying off your children and ensuring the family line survived, was a major ordeal in many parts of the country even when you had plenty of food, because the production of that food in Norway, even when the climate has been "good", has meant the farming part of the population has been spread out far more than many places.
There were no major cities. By the 1600's, Bergen, now Norways 2nd largest city, with 285k people, was by one source described as "definitely the largest city in the Nordics" with ca. 15000 people, as a result of being a major Hanseatic league trading port, but while it was larger than e.g. Copenhagen, Denmark (and Sweden) had many other cities of some note, Norway did not. The old capital, Trondheim, by then had reached 5000, and the other current cities in Norway had far fewer.
Even today there are parts of the country which if you avoided car travel would seem almost as desolate as in LOTR, and the population density today is more than 10x higher than before the Black Death. In the 150 years following the Black Death, you'd have walked through a landscape that would have appeared mostly untouched by man, with the very occasional farm, sometimes days apart, which half the time would be in ruins, and the very occasional small village.
If the population density was low to start with, and food production tough, recovery from even a single major enough event can take a very long time.