What is it
A "visual library" would be a collection of photos, painting, graphics, patterns, artwork and colors that could be kept in your head, in print, or digitally.
Working on it
You can work on it by viewing and studying as many photos, graphics, patterns, artwork and colors as you are able. And I mean, really study them.
Think about why someone took the time and effort to create that piece, why others may like it or hate it, what emotions do you feel when you look at it, ask other what they feel, why do you feel that way, what changes would you make to it, would your changes make it "better" or "worse", and anything else you can think of.
As a photographer I spend some time everyday looking at other peoples photography and read what people have to say about them. I try to take the time to really think about what makes each photo appealing or appalling.
Why is it Important
It is important to have a strong visual library so you can know what parts to steal and what to avoid. So you can recognize what makes something "good" versus "bad", and to develop your own sense of taste.
I think it is necessary to developing a sense of who you are as an artist and who you want to be.