Though it is a pretty novel idea, I think your audience matters a lot. I would think very few people will get the variation on ′ and '' for denoting time. A few nerdy engineers and scientists might, and they will likely enjoy it and giggle in their beards.
Here is the problem: ′ which is not the same as ' are usually used to represent feet. To be more precise, it can represent feet, archminutes and minutes.
The '' which is not the same as " are usually used to denote inches. To be more precise, it can represent inches as well as archseconds and seconds.
(NOTE: these symbols might have different significance and meaning depending on country!)
3′ 5″ could mean 3 feet and 5 inches (of length), or 3 minutes and 5 seconds (of time). As an angular measurement, 3° 5′ 30″ means 3 degrees, 5 arcminutes and 30 arcseconds.
As for the degree symbol, though it is a nifty idea, again, I think few people will get it. "4degrees until this thing is finished"? Whhhaa?
You could pull it of, but this is heavily dependant on your audience, and you MUST have some visual clues along with it to get the "dimmer" people on board. Then, if you ask us what visual clues you could use in combination with your degree symbol, you might get some interesting answers. But I doubt it will be wildly user friendly.
Time is traditionally described with 02:34, or if you would be extremely clear: 00:02:34 to make sure people do not think that this will take two hours and thirty-four minutes.
Again, depending on your application, this is the normal way programmers do: 2minutes, 34seconds. And for being solid, it would detect anything below two and say "minute" and "second" for single values.