We have been automating Adobe InDesign for 12 years, InDesign Server for 7 (since it came out), and it seems that just now we are finally being proven right in our bet long ago that going with the slowest, yet far and away most robust, composition engine would in the long run be the best path. After all, hardware just gets faster and faster. Desktop tools and capabilities do tend to end up on servers.
Now, in 2012, we can finally say that advances in Hardware, RIP software, and InDesign Server itself, have given us many tastes of sufficiently fast performance, and we can see a vision when InDesign Server is the tool of choice for even the most throughput-intensive documents, such as transactional statements.
Back when InDesign Server was a wild and crazy idea
In the year 2000, it was quite a long-term vision. Having just started Silicon Publishing, we wanted to do exactly what we had done previously at Bertelsmann: compose data-generated documents like catalogs and directories from data sources, flowed through document templates. At Bertelsmann we had primarily used Xyvision, which was an industrial-strength composition tool with powerful automation, robust page composition, and stunning throughput. Yet the cost was prohibitive to our small, new company and we weren’t eager to hire a “xy operator” who knew the esoteric ins and outs of making the thing work. There was something liberating about controlling the complete job from one piece of code, and seeing the output in real time in a desktop application like FrameMaker or InDesign.
Those were the two tools we tried: FrameMaker 6.5 and InDesign 1.0. Frame had MIF, a wonderful language for defining documents, and we got quite adept at generating MIF from code, even from XSLT in some cases. With InDesign it was a matter of using a scripting language: AppleScript or VBScript at the time. We later moved to ExtendScript as our primary language for automation, which we now augment with C++ plugins and/or HTML5-built CC Extensions as appropriate.
Frame was much more robust, especially back then when InDesign didn’t even have tables. But Frame was old, while InDesign was exciting and new: the exposure to automation was unparalleled, and as we saw the momentum of the product we could feel that this was going to be the place Adobe put their best typography and layout capability, and we knew Adobe was uniquely qualified to see this through.
InDesign Desktop Did the Job Already
We got work automating catalogs and directories, and when we provided data-generated typesetting as a service, nobody knew that we weren’t using Xyvision or something comparable. We found we could automate InDesign to do pretty much anything we could do in Xyvision. It was sometimes a pain to automate things that really belonged in the product (tables didn’t show up until InDesign 2.0) but the momentum of the product with each release felt wonderful, and the fundamental full exposure to automation was a joy: anything we could imagine, we could do. This could not be said of any previous layout application we had seen, nor any we have seen since.
It was a bit ridiculous, however, when we would deploy automation solutions to our clients. At Bertelsmann we had mainly been a service bureau or ASP-model solution provider (what they call SaaS today… in the 90s we did have SaaS, Cloud, and AJAX, by the way, they had merely not yet been named and hyped by the late-adopters). With a hosted solution we could turn 300-page complex directories in seconds or a few minutes, depending on complexity. Tolerable speed in any case, from a true server.
As a small company, we could have provided hosted services, but (a) at the time we weren’t even perceived as a company, we were seen as consultants because there were so few of us, and (b) the tools we were using weren’t yet licensed for server usage. FrameMaker Server did come out soon after we started, yet InDesign was not licensed to be used as a server nor would it be practical, though we did hear of companies setting up “desktop farms” some of which we know persist to this day.
Yet our client base grew, and many wanted to run automated publishing themselves, in house. In such cases, we had to explain to them that they would need to set up a desktop workstation, and in some cases (i.e. producing 150 100-page directories driven by a business rules table) let a process run overnight: they would literally see the documents compose on the screen. They scratched their heads, but they didn’t complain. Many of these solutions have run for years and we still have clients that run old desktop code.
Still, it was surprising how solid the applications were and how much throughput moved through them. We found InDesign kept getting better and better, and the precision was stunning: for example we automated generation of patient encounter forms, and we could analyze proximity of black ink to the bubbles that users would mark, and we could define proximity rules and copy fitting logic in composition that allowed maximal usage of the limited space on these crowded forms. Automation precision hit new heights as performance hit new lows.
What part of “Server” didn’t Adobe comprehend?
It was just a bit silly that this was not on a server. We had been doing work for Adobe since 2000, yet we weren’t deeply connected to the InDesign group. We begged anyone who would give us the time of day for an InDesign Server, and it gradually dawned on us that Adobe was a collection of largely un-connected product silos, so whatever emotion we brought to bear on a doorman or an Acrobat Sales Rep was unlikely to have great impact. One day, as if our pleading cursing and begging had made an impact, we heard of a “Server Tools Group” and the advent of “Altercast” and “Document Server” – Illustrator Server and FrameMaker Server, if you will: we knew there was another shoe to drop.
Sure enough, completely unrelated to our impassioned pleas, in 2004 we finally got wind of an effort to create an actual InDesign Server. Based on InDesign CS1, there was an internal project that split out the model from the UI and allowed for server-like usage of the core InDesign engine – notably, this also had XML round trip that was prototyped but never showed up in the product. In 2005 when I got a chance to talk to the people in charge of the InDesign Server project for CS2, I said that we should definitely be in the beta as I was probably the only person in the world who literally dreamed of an InDesign Server. We were included in the first group of solution providers/resellers. Yes there is a God (I think her name is Whitney McCleary).
InDesign Server is born, slow as molasses
We got a bit of work for Adobe writing about the release of InDesign Server, and I wrote an article for InDesign Magazine about our love of this new thing. Yet when I went up to meet the great Whitney McCleary, one of InDesign’s spiritual leaders, I was given the bad news about performance… it was still slow. “We decided to release this technology to developers,” she said, “and we decided not to support it.” There was extremely little server about InDesign server, they had essentially hacked out the parts that didn’t depend on UI and wrapped them in a little bit of a server-esque container so you could send a SOAP message that would tell it what script to run. You could spawn multiple instances, but the rewards were not nearly linear, even a fairly decent server would not stun us with throughput. Initially the license was antagonistic to SaaS or virtualization or anything like it, clearly this was tossed off with caveats that indicated Adobe’s remaining bias against servers, webs, interwebs, or the like.
So here was the real bet I’m talking about. Five years had gone by, we weren’t impressed with speed, and were disappointed that the software itself wouldn’t help us much with making it go faster. Yet the quality… by now, there was pretty much every bell and whistle you could imagine: Transparency, tables, text on a path, referenced objects from Illustrator and Photoshop, really high quality PDF output with fine-grained control, I think by CS3 the feature set was at the point you could say it had a far-and-away advantage over any other page layout engine. Yet it was… slow.
At least we had a server, we had a mechanism to add servers, but for transactional documents and even some classes of promotional documents it was prohibitively slow. We used XMPie and built our own plug-ins to do what XMPie did, but even that was slow. And the documents created could often take forever to RIP. We liked XMPie’s export to VIPP, PPML, VPS, etc. yet there was still no comparison to a native VIPP job or old-school AFP.
Why was it so slow? Mainly because it was doing substantial processing: no pain, no gain. Managing layers of transparency has obvious processing cost. When you use the Paragraph Composer, it is not the dumb, brute-force, “slap text on a page” process of most transactional engines, but the artificial intelligence that makes InDesign output look like it was crafted by a seasoned typesetter requires that the content of the entire paragraph and all contained styles are analyzed to determine how characters are placed on the page.
We dabbled in faster, less robust tools but decided fundamentally to stick with InDesign, as by then it had completely differentiated itself from any other composition tool both in terms of quality of output and exposure to automation. Our logic in sticking with InDesign Server was that some day, hardware would reach performance levels where throughput would be acceptable for more and more document types, and that PDF-based RIP software would one day work.
Sure enough, doubling every year works wonders for speed
Having been heads down the past 5 years deploying InDesign Server solutions, I look around today and I’m happily surprised that our dream is starting to come true. We are not benefitting from a throughput improvement on quite the scale of Moore’s Law, as processor speed is just one factor in IDS throughput, yet we are finding new multi-core servers are exponentially faster than what we used to see and more and more document types are becoming viable with IDS. And PDF-based RIP software is truly stunning. HP in particular has been fantastic at building out RIP software and Adobe has made some steps towards optimizing InDesign Server for high throughput applications. The beauty of this to us is that the page layout capability is not sacrificed at all; there is less and less need over time for compromise between throughput and output quality.