My son (13) is taking a look at some of the stories written by my customers, as part of the lit. crit. syllabus of his home schooling.
Among his various insights was that people didn't seem to know what year it was; instead they say "it were nobbut three summers gone" or "he'll be of age two winters hence". Even after I pointed out that the stories were set in a fantasy world more or less analogous to Medieval Europe (aren't they all?*), he insisted that people would have known what year it was even if they weren't themselves literate.
The convention of "rural folk" using "charming" expressions like these seems to have been a largely nineteenth century invention (I find very few references to it before then, but it seems to explode after Thomas Hardy. I've searched the text of four Hardys without finding it in them, yet books that are described as "reminiscent of Hardy" all seem to contain it!).
Indeed we have historical and archaeological evidence that people have known what year it was going back thousands of years.
Now if you are writing a fantasy set in a world you created, this little bit of poetic licence might be apposite if you really can't avoid referring to a specific year (for which you would have to go to the trouble of inventing a calendar), but why wouldn't you just say "three years" ?
The word "year" itself appears to be unbelievably ancient and almost unchanged. Doug Harper shows that the word is common across many languages, modern and ancient, and may have originally been an idea representing a "complete cycle" – a frankly more probable and useful image than that of summer's lease – as evidenced by the durability of the word.
Like many literary conventions, writers have got so into the habit of seeing "twenty summers" in the rural/historical genre that they start to feel uncomfortable with "twenty years". You should not. It has a long and distinguished pedigree, and a power and meaning that is more complete and more significant that the twee poetic charm of any individual season, so you should claim back the word "year" and use it with pride and confidence!
* at least two of my customers manage to deviate from this norm, so no, they aren't all.