This isn't really an answer, but I thought it might be easier to read in answer form than in comment form. One of the key things to remember about RGB images is that they are a grid of pixels, each square takes 1 byte (a value from 0 to 255), and there is one full grid for each color.
the grid size is the dots per inch (pixels per inch) multiplied by the final print size. Your 4x4 foot image:
48 x 300 = 14400px
14400 x 14400 = 207360000px (area of image)
207360000 x 3 colors = 622080000 bytes = 593MB
So your minimum file size is going to be half a gigabyte. Each layer adds something to this. If you duped a full color-flooded layer, you would see a doubling, but in my work, most layers are small fractions of the overall image size, and photoshop only needs to store the smaller bits.
What this means is basically what Alan said: ample RAM memory is crucial to working with these images at 300 dpi. You don't mention OS, but a 32-bit windows machine is pretty much maxed at 3GB of RAM. To put more RAM in your machine you'd need to know the OS can address it and your motherboard can accept it.
Of course, the more data there is to manipulate, the longer the CPU has to work to make that happen, so processor speed is a factor, but memory strikes me as a limiting factor in this case. Once the actual RAM is full, the OS has to start switching RAM data to the hard drive which is at least an order of magnitude slower than RAM. Which means go get some coffee while it works.