The main limiter on your ability to conjure is Drain dice rather than Summoning dice, so even with only 1 magic allocated to casting, you can be a pretty decent summoner if you have a solid drain stat.
That said, it's never a good idea to ignore either spellcasting or summoning on a magician or mystic adept.