How to achieve woven displacement effect in photoshop

Question

I want to replicate a woven effect of socks in photoshop. But not only with importing a texture and putting my design on top, I want the actual design to be displaced (see my refernce image) like it would in real life, due to the stitch.

This is how the design looks like on the socks:

And this is my current state by using the mosaic filter:

There are a few problems with it: It's just pixelated and it doesn't follow the real structure of my stitch. But thats no so much of a problem when looking from a distance. The bigger problem is that the color isn't exact black anymore and my design now has half transparent pixels (which is not possible on a sock obviously).

Also coloring it on top is no option, because there will be multicolored design in the future, this is just an example.

I tried to use a displacement map, but had no luck with it (anyway I think it's the wrong approach).

So what do you have an idea how I can get close to the real woven effect? I couldn't find any tutorials about this topic.

Important for me is, that it should work on a smartobject, so I can exchange the design without any big adjustments. So an action would be possible if there is no other way, but I think an action can lead to more errors because I have to keep imagesizes etc. in mind.

Thank you very much.

Edit:

This is my result with a displacement map:

And this is the "map" I used. Just a photo of a white sock with some contrast adjustments.

And if it wasn't clear, this is my original vector design:

Edit:
I tried user287001's solution. But the problem is I face the same issue as before with using my displacement map. I think I use it just wrong. The line in your "design" is still connected and it will just be displaced.

In my case the design will get split apart into lines. What did I do wrong? Or what settings did you use for your displacement?

My design:

My background (I know the seam is bad):

Displacement map:

My result:

Answer 1

This knitting style needs a displacement map which distorts like there's ridges. To get it I drew some lines on a piece (there's no big piece free) of your fabric. I used red to see it clearly. It will be recolored later:

It's turned to black, merged with a white layer and blurred. The contrast is stretched to cover all grays between black and white:

The displacement map is saved. It will be used to distort my own art layer:

A few trials were needed to find the good horizontal and vertical displacement amplitudes. The result was finally this:

The sharp edges are not plausible. Blurring fixes it partially:

I guess also a little transparency would be good to show something of the knit texture. The displacement map could also be a mix of my version and the texture. I skip it.

This case was quite easy because the ridges are so dominant and I have drawn my "artwork" by paying no attention to which directions the black lines are really possible to be knit. It's now like painted, not a part of the knit structure.

Different knit structures need different treatments. I'm afraid something which knows the real 3D structure of the fabric would be generally needed for more plausible results. There's already suggested to check a textile design website. I must agree. With no knowledge of real textiles nothing really plausible will be born.