So the reason i'm designing in Photoshop instead of using vectors is because i am incorporating photos into the designs. The shirts will be printed using DTG Neo Pigment. What steps should i take to ensure higher print quality? Is there any specific color mode to use, or what file format should i use? Please advise and give any tips, thank you in advance.
You can well design in Illustrator if Photoshop's vector tools are not enough. Vectors can be rasterized in Illustrator or you can copy and paste them to Photoshop, which can be useful because many things have better tools there (you said you use also photos).
Best quality: The sharpness and colors are limited by the printing process. It's useless to try more than the printing machine + its software are capable. The printer asks raster images because it is the shortest way for him. It doesn't spoil the quality, because finally everything is rasterized. Plotters and cutters use vector images as the final file format.
Your printer surely tells which is the biggest useful image size in PIXELS. Stick with it. In addition the printer can tell the most useful proof color setup - no matter you work in RGB to make the PNG. Then you can get the best idea of final colors. They can also have some guidelines or examples which utilize their process to the limit. Ask!
If the quality is bad now it can be caused by you or the printer. His process can be wrong for your expectations. You should contact them and explain what's the problem (we cannot guess it). Otherwise no fix is possible.
If the printer doesn't give proper answers you are in a wrong place. Serious business wants to keep paying customers.