It's debatable.
Astral Combat is a big fuzzy mess because it gets less than one column of one page, and is largely summarized by, "Astral combat is resolved in the same way as physical combat." That summary opens as many questions as it answers - are there bonuses for cover or movement? are there environmental penalties if the astral space is complex? is it more like melee combat or ranged combat? can it really be compared to physical combat when it is entirely psychic, involving only Mental attributes?
Back in 4E I had a messy astral combat sequence where the players won by spamming a superior opponent with 5 watchers. The watchers only had 1 or 2 dice to roll for Astral Combat, but their accumulated attacks whittled away the defender's dice pool because of the "Defender has defended against previous attack" penalty, which I applied. Maybe I shouldn't have, given the inconsequential nature of being attacked by a watcher, and this certainly isn't an issue in 5E, where watchers are expensive and rare, but it made me rethink how literally to take the "Astral combat is resolved in the same way as physical combat" statement.
Given the uncertain nature of astral combat, I'm inclined to keep the dice pools as straight and clean as possible, with relatively few modifiers.