How to edit text that is in a jpg photo (using newer technology)?

Question

I know that there is some software that makes it possible to instantly replace text in a photo, as shown here (from this TechCrunch article about a feature in Google Translate released in 2015).

If I have Photoshop or some similar software, how can I select an area of the photo that contains text and then make it editable such that I can type new text that will appear in the same style as the original text?

P.S. My question is very similar to this one but I am already fully aware of how Photoshop files and layers work and how text shown in jpg photos isn't directly editable. I'm thinking there is probably a new tool or feature available sort of like Edit > Fill > Content Aware.

UPDATE (in response to comments):

Look at this screenshot of my iPhone's Google Translate app translating this StackExchange page from English to Spanish on the fly:

What you're looking at here is not a webpage. (Yes, Google Translate does have a webpage, and it's totally easy to understand how to replace text within HTML. This is different. We're looking at an iPhone app that manipulates whatever live image my iPhone camera is currently looking at. It's augmented reality.)

It's pretty amazing, and it seems to be better than rudimentary "drawing a rectangle over the text in the same color as the background and creating the text again". In fact, if I hold my phone camera at an angle, it's able to replace the text at that same angle (and color, etc).

And here are a couple other great examples:

Answer 1

Automation of nearly anything you can do semi manually. Most is possible to do if you have a dedicated application development team*. The tools that are need to do this have existed for quite a while all you need is:

• OCR (Optical character recognition)
• Content aware fill
• Translation service

It so happens to be that google has all of those components. Most likely you also have the OCR and content aware fill available as separate programs. It would be possible for you to build your own app that does this. However outside the realm of machine translation for humans on the move it is of relatively low demand. In fact the first 2 stages are pretty easy all you need is the bounding box of the OCR result. Adobe has all that needed for this (Acrobat has a pretty decent OCR that even can recognize my hand writing).

So why not? Its not terribly useful for a graphics designer. I mean I have done jobs like this before. On the other hand, going back to original designer and ask them to modify the file is easy. So you have to look for applications that are geared to persons who can not afford or do not want to pay for graphics workers to get their work done.

So there is the question when some tool is actually useful and when its just a novelty.

* In fact 47% of all jobs are under threat of automation according to a Oxford study. So you feel lucky.

Answer 2

Keep it simple. Try Photoshop's content aware deletion of the text area. Not definitely the latest automatic text recognition technology, but helps to change the texts. The result may be perfect or easily fixable if there exists nothing difficult to guess under the text.

When cleaned, add a new text onto the photo, if needed. Unfortunately (fortunately, methinks) the style of the new text isn't automatic.

Start by placing the photo to the background in Photoshop. Then select the text as tightly as possible to minimize the area that should be quessed.

Draw a selection rectangle or polygonal lasso around the text.

Tighten the selection; use Select > Color Range. Don't add-pick the hazy color at the edges of the letters, rather expand the selection a little to have the letters selected. Nearly transparent watermarks are often hard to select, but I am not talking about how to remove watermarks.

When the selection is good, press Delete and choose Content Aware. I have surprised how well it quesses and how little manual fixing is needed in easy cases. Case is easy, if the text stands over uniform color or there are mainly straight borders or large smooth textures under the text. I don't believe that somebody's face will be recreated under the text.

Answer 3

To answer your question

how can I select an area of the photo that contains text and then make it editable such that I can type new text that will appear in the same style as the original text?

would be: In a program for translating pictures that contains text.

Photoshop is programmed to do one thing "manipulate/create images".

OCR programs are programmed to do one thing "OCRing text"

And translator are for translating.

So to have a program that do all the three things you have to sacrifice some application of before mentioned program as not needed (ok, who in the right mind need the 3d function in PS?) to focus only on the "filling". The content aware part is moved to the OCR Part that should happened as step 1.

Why there is no such option in Photoshop? Because... why would you need it there? If you use PS you should also buy and use Acrobat. And if you need to translate a big chunk of text you should use translation bureau.

Those nice pictures you showed are very nice translated because they are simply, short and on one colour background with sansherif font. How do you think it would compete in translating german Lebensabschnittpartner in Times on rainbow gradient?

Your picture of google translate show how badly really the software works in case of looks ( it could assume that the block of text is written in the same font, same size and on one grid) and if it's GT I think the translation job is not the finest one.