Generalizing a bit, but one of, if not the most important factor in the N64's poor performance was the fact they stuck with cartridges. All the best 3rd party developers went with the CD based PS.
"The mass exodus to Sony was huge. Everyone and their dog began making PSX games. Nintendo and Sega's response? To get more ruthless with their ideals. "Deal with us only, or suffer!" Well, that was it. The developers weren't coming back. In 1996, so many people wanted to make PSX games that Sony limited the market to 150 titles that year as the installed base wasn't that high and they wanted everyone to make money. (oddly enough, the really bad games that didn't make the cut ended up on the Saturn, which explains the exceptionally bad games for the Saturn that year. Plus, it damaged the image that Saturn had too as all the "exclusive" games on it really bit.)"
Taken from:
The Gord Speaks