We have created an Indesign document for 450.000 letters we need to send. Included in this letter is a Datamatrix barcode. We are trying to merge the data, but it takes forever, we can only do batches of 1000 or Indesign starts choking (no hardware issue, there are low resources used). What kind of software can we use or buy to do this job based on our Indesign template, data and barcode image file with a (much) faster speed?
We are on a tight schedule so we are looking for a fast solution.
You can operate on the PDF file, there are software that can merge pages in a PDF. This is fast as its basically just a append operation to the PDF file. (you can do this with a text editor if you must). Off course you lose the features of inDesign.
On a similar note you can let the printer add the changes, or inject the change just before print, by including the data matrix in the PostScript file itself.
The real question is could in design be made faster? Most probably the reason your merge gets slow is that you try to fit all in one pdf file. If you chunk the operation to about 250 letters per call then it would probably work out fast. So just chunk your data into pieces. Open each piece into a separately in inDesign export and move to next piece.
If you were to do this manually it would take ages. But even if you have not scripted before it takes less time to develop knowledge into scripting to get this done than doing it manually. It is after all not more than a for loop.
Anyway this is more of a question to Superuser.
PS: Might be a good idea to hire somebody who knows what they are doing though.
InDesign indeed is not performing enough with such huge documents. Once that said, you can think of a script that will generate documents one record at a time. But given the extensive amount of documents to produce, you may consider PDF/VT solutions like DesignMerge from Meadows Publishing, XMPie from Xerox, SmartStream designer from HP or Callas PDFChip. Generally it consists in having one PDF template and the data aside so the print engine is able to print all the customized files from the template. We have a great experience of such processes. Feel free to get in touch at ozalto.com
You are all wrong, the software is out there. I work for a bulk mail company and routinely export PDF files through a plug-in designed to interact with Indesign. I still have all Indesign capabilities and can attach lists that are up to 100,000 names in as many merge fields as i want. The software is called EngagIt. I can do everything from merge imagery, to merge text, even custom art for each person on my mailing list (i.e. Color changes, photo changes, background changes...ect.)
Word and excell? Even excell has a limited table capacity, so split the data.
You also can try a server side script like php and use mysql to store your data.
But I have the feeling that the Datamatrix barcode is the one that is slowing the process, if it is generated on the fly.
Use InDesign only to export a static document as pdf and then use a professional vdp tool such as PrintShop Mail (standalone software) to bring up your variable informations (images, text, barcodes, ...).
