InDesign Data Automation for Books, not Catalogs

Question

I have a use case where I’d like to use a bunch of small pieces of data from a database, in random places inside a large book. My book has hundreds of photos (with captions and credits) and over a thousand references (this kind of thing: Atkinson, J. W., & Rayner, J. O. (Eds.). (1974). Motivation and achievement. V. H. Winston.) All of the data for both the photos and references are currently being stored in a database. In a perfect world, I’d have a panel in InDesign with a list of all the image IDs and with a click, the image and caption would be inserted to the document, I’d have another panel with a list of reference IDs that would have the same functionality.

Here’s what I’ve tried so far:

1. After reading about how to import XML data and maintain the link to the data file, I fantasized that I could just drag the data I wanted from the structure panel, to the spot in the text where it belonged. This obviously moved it under the story’s node and broke it’s connection to the XML file.
2. I've downloaded and worked with the trial version of: DataLinker, EasyCatalog, InCatalog and InData. These all seem to be centered on variable data printing (catalogs, personalized mailing, etc.) and not on placing bits of information, randomly throughout a text document.
3. Given my background in web development, I studied a bit on ExtendScript, and pulling from several scripts that are around, I made a script that will import text variables from a text file. I can easily export my references and image data from the database, I planned to use GREP styles to format the reference data. The issues with text variables are described very well over here (unfortunately I didn’t see this article till I’d finished my variable import script.)
4. I also tried using a script that utilized Bridge to load data from a locally hosted webpage that serves the formatted info for a given image or reference, based on a URL parameter. The script to over 30 seconds for each call to the webpage. Also, I didn't have a way to keep track of where this was done, so it could be updated if the database changed.

Any examples on how to accomplish this, or articles around the web that I might've missed where someone discuses this usage of data in InDesign?

Answer 1

So I'm having to eat my words about EasyCatalog not seeming like a good fit for what I needed. After watching some of the videos on their YouTube channel, I started seeing how simple it would be to get the project rolling with their plugin.

Here's what I've done:

1. Exported the required data from MySQL to a simple CSV file, one for references and one for the images.
2. Import the CSV files into InDesign.
3. For the references I specified that the content field has HTML embedded in it, most of the references have a few words that are italicized and they came out of the database wrapped in <em> tags.
4. For the images, I set the filename field to be an image field, specified the folder where the files were located.
5. Setup a simple InDesign Library file and included a basic image with caption field, tied the image and caption fields to their respective fields in the data source.

Now, all that's needed is to put my cursor where I want an inline image and right click the library item, choose Place Item(s), then drag the desired image record over the placed block—Boom, there's my image and caption, with a link back to the CSV file, so when I re export from MySQL and overwrite, the data is updated in my layout!

For the references, I simply drag the field (not the whole record, like described for the images) to the spot I want it. Italics is maintained and the data is updated properly.