Design is a specialized job that requires specialized skills. If you are lucky, you have access to professional designers and user experience experts. However, keep in mind that even those professionals probably have little or no experience designing holographic worlds. It is up to you to tell them what to design, how to design, and what things to keep in mind when designing.
I strongly believe in the Agile approach when it comes to developing software. This means that we form teams that are fully self-sufficient. In that philosophy, designers are part of the team, just like the developers, testers, and other professionals. Design is a very important part of developing software, and more so in a graphic-rich environment, such as apps for the HoloLens.
That being said, not all projects need graphical designers, user experience designers and, interaction designers...