The difference comes from having two common understandings of the word "kilobyte".
The SI prefixes kilo, mega, giga all refer to powers of ten. A kilometer is 1000 meters. However, in computing it has been common for people to say kilo when referring to 2^10 (1024). Mega is used for 2^20 (1048576). For KB, that's close, but the higher prefixes things get further and further from the non-computing meaning.
For a long time this has caused confusion. Hard drive manufacturers have long used the base-10 values to describe the size of their drives, which can upset someone when their "500GB" hard drive shows up looking smaller on their computer, if it reports the base-2 values.
ISO standardized separate prefixes to use when discussing base-2 numbers, in ISO/IEC IEC 80000-13:2008. Instead of kilo, mega, and giga, we have kibi, mebi, and gibi. The standard abbreviations are KiB, MiB, etc.
Why does this matter?
Photoshop is reporting values using the base-2 values. OS X switched to base-10 sizes in Snow Leopard. (You can switch it back if you want.) So 2KB in Photoshop means 2048 bytes, which is 2.048KB in the Finder.
There's still a very small discrepancy between the value that PS reports that I believe is due to the extra metadata and header flags that PS adds during save.