If you are saving the document as a Photoshop PDF with the "Preserve Photoshop Editing Capabilities" option checked, it will also include the Photoshop document.
If that document has multiple layers it could be what is making the file size so large. So you might be able to get the file size down by simply exporting without that option checked.
Here's an extreme example: for a piece of work I was actually commissioned to produce for a client.
I had a PSD document which was 299MB - with many layers. When I saved as Photoshop PDF with the "preserve photoshop editing capabilities" checked, the PDF file size was 306MB. When I switched that option off, the PDF was only 17.4MB
Of course, this will depend on the kind of image you have, but try that solution, and see if it changes the PDF size for you.