Is there a way to determine how large a specific Photoshop layer or group is in terms of bytes?
There is usually an image size indicated at the lower left of the image frame (e.g. "Doc: 1234.3k/2345.6k").
Frankly, I do not recall what the leftward number indicates, but the right-hand number is the uncompressed byte size of the document (in memory). A straightforward way to tell what a layer size is: write down the right-hand number, delete the layer, write down the right-hand number, undo. Then subtract.
Note this is the size in memory, and is not the same as the file size, since most storage mechanisms compress the data in some way.
The layer size will vary based upon coverage. A single-layer image 100% filled is roughly the pixel width x the pixel height x the number of channels (RGB=3; CMYK=4; alpha=variable). A partially filled layer will probably be less than this and is at most the dimensions of a bounding square around the non-transparent areas. If I had to guess, I would expect photoshop to optimize this in some way by using multiple bounding boxes (behind the scenes)