troubleshooting: the two laws - ixiasoft user conference 2016

Troubleshooting the DITA CMS:

The Two Laws

Alex Kozaris, IXIASOFT IT Specialist

Bio Alex Kozaris • Computer geek since 1985 • Tech comm since 2005 • Peartree 2005

• Writer, designer, web dev

• RIM/BlackBerry 2006-2015 • Writer, CMS support and admin

(XyEnterprise Contenta & IXIASOFT DITA CMS), Tools & output dev, Localization, IT

•  IXIASOFT 2015- •  IT (deployment and support)

Agenda

• The Two Laws • The Three Understandings • Troubleshooting Example • Last Resorts • Q&A

The Two Laws of Troubleshooting

Murphy’s law: If it can go wrong, it will… There is no way to completely fail-proof any enterprise application, especially involving customization, because customization is purpose-built on the fly, and extensively testing this is impossible The more complex a system is, the more opportunities there are for things to go wrong


Don’t worry, Chicken Little. It’s just a piano.


Occam’s Razor All things equal, the hypothesis with the fewest assumptions (often read as ‘simplest explanation’) is best. Always start with the easiest explanation (with the fewest moving parts), then work your way up in complexity.


House, M.D. – an American medical TV show (and fan of Occam’s razor)

The Two Laws of Troubleshooting Taking both of these things into account; the best hypotheses to test are the ones that: (a) are most likely to have gone wrong (b) require the least effort to test

The Three Understandings

The process is simple: you just need to understand three things. • Understand the system from the user’s perspective • Understand the flow of data • Understand your logs

Understand the System from the User’s Perspective •  People with a technical background often

see computer systems as hardware, software, data, variables, and code; not as a user-centric experience

•  Learning a bit about what the users are working on will help you understand what they’re trying to do

•  If your users do a lot of output generation at the end of sprints, you know when to expect heavy load and performance issues

•  If your writers use linking heavily, look for unresolved or out of scope links

Understand the Flow of Data

• Knowing how the data moves through the system will allow you to hypothesize on what could happen to it along the way.


•  Imagine a child came up to you and said, “I put my sailboat in the river and I waited for it at the end! it never came down! Can you help me?”


•  If you know how the bends and bumps in the river, you’ll know where to look for the kid’s sailboat.

Understand Your Logs - TEXTML

• TEXTML Server § TEXTML Server Log

§  C:\ProgramData\IxiaSoft\TextmlServer43 §  Always start here!

§ TEXTML Log Calls §  Transaction log §  Usually only enabled for troubleshooting or

performance monitoring

§ Windows Event Log §  System-level issues that might have caused

TEXTML Server to crash

Understand Your Logs - Client

DITA CMS Client • Dita_cms.log (debug mode) §  In the workspace § Run eclipse.exe –debug or startcms.bat with debug.txt in

the root of the workspace • The log window (export to tsv)

• TEXTML Server log

Understand Your Logs - OG

1.  Wrapper.log For the outgen service

2.  Build log For your particular output (HTML)

3.  DITA OT log From the DITA Open Toolkit

Understand Your Logs - OG • Output Generator logs look scary, but really aren’t

that bad. You will get the hang of it. J • Compare a “good log” to a “bad log” • Good words: § BUILD SUCCESSFUL

§  Doesn’t necessarily mean everything is perfect, just that the OT didn’t encounter a fatal error.

• Bad words: § BUILD FAILED § Exception § Error §  Invalid

Understand Your Logs - Web

Web Author & WCR 1.  Tomcat log

<TomcatDir>\logs 2.  Glassfish log

<GlassfishDir>\glassfish\domains\ cmsappserver\logs

3.  TEXTML Server log 4.  Java console log

Case Study: Output Generator

User’s complaints: 1.  My PDF didn’t work. 2.

Understanding the User in this Case

Understand the user’s perspective: § User wants to build a 60 page deliverable for a new

product § That product has 3 major components, and the

deliverable is broken down into three submaps, one submap for each component

§ Some topics are shared with other deliverables, and are differentiated using conditional publishing attributes

§ One submap has 15 graphics in it; the other components of the product do not have a graphical aspect

§ Output type is a PDF for print

Understanding the Data Flow in this Case

Understanding the Logs in this Case

Look for information in the following logs: • Build log • OT log • Wrapper.log

Going Through the Data Flow...

1.  User chooses Generate Output from the right-click menu. §  We know this happened, because the user said their

PDF didn’t work. This means they selected their output format already.

2.  CMS retrieves output types from the OG. 3.  User selects output format and ditaval (if applicable)

(outputtypes). §  Again, we know this happened.

4.  CMS sends the user-selected info to the OG.

Going Through the Data Flow...

5.  Output Generator creates temporary directory for the build and retrieves the files from the TEXTML Server.

6.  The OG calls the DITA OT to create the output (ant –f build.xml)

The DITA OT creates a .fo file and calls the rendering engine to build the PDF.

7.  The OG returns the zipped output to the user.

Applying the Two Laws

Murphy’s law—what could go wrong?

Occam’s razor—what requires the smallest ‘leap’ to envision?

The service crashed, terminating the process

Possible, but not likely; if this happened, no outputs would be running (and you’d have heard the issue raised by multiple people no doubt)

There was an authentication issue with the service

Also possible, but not that likely either, unless someone changed the password or disabled the service account on the domain, which is highly unlikely

The disk was full Quite possible; running out of disk space would cause any output to fail

The TEXTML Server was not responding

Unlikely; the entire system would be down, and you’d know about that too

Looking at Processing Task Flow 5.  Various processing tasks are run on the files

(preprocessors > outgen-init) Murphy’s law—what could go wrong?


The preprocessors.xml file could be corrupted.

Unlikely; unless there has been a recent change to one of your formats, and this is the first time someone generated an output using it, other formats would also be broken by the error, and this wouldn’t be possible because you tested it in your development environment, right? ;)

The ant code could have a problem in it somewhere. (e.g. the preprocessing instructions could call an invalid task.)

Slightly more likely; would affect only this format, but still likely wouldn’t skip through a round of testing

Next Steps 6.  The OG calls the DITA OT to create the output (conductor)

§  The DITA OT creates a .fo file and calls the rendering engine to build the PDF.

Murphy’s law—what could go wrong? Occam’s razor—what requires the

smallest ‘leap’ to envision? There could be an issue in the content that prevents the .fo file from being generated

Plausible. There are many (obscure, but possible) ways to create content that is valid XML yet breaks the DITA OT during transformation.

There could be an issue in the XSL stylesheet.

Plausible. If an issue was specific to a certain DITA element that had not yet been tested since the last stylesheet update, the issue could rear its head here.

The .fo file could be invalid. Plausible. If a user created a scenario that was valid XML and valid DITA but not valid after being transformed into .fo, the PDF generation would fail.

Font(s) are missing. Plausible. If a font referred to by the XSL stylesheet is not installed, the PDF will fail.

Problem with the Output Generator?

7. The output generator zips the output and returns it to the DITA CMS client

Murphy’s law—what could go wrong?


The PDF could not be zipped or returned due to an error.

Not likely; even when an output fails, DITA CMS will usually give you an empty zip, or a zip with only a log file in it.

What are the Plausible Explanations?

Let’s make a list of the explanations we considered plausible: 1.  System is down/service level issue. 2.  The disk was full, so the files were not retrieved

from TEXTML. 3.  Preprocessing issue (ant code). 4.  XSL stylesheet issue prevents .fo from being

generated. 5.  Content issue results in invalid or missing .fo file.

How Do We Verify?

1.  System is down: Generate an output that’s known to work.

2.  Disk full: Test #1 would eliminate this one too. 3.  Preprocessing: Generate the same output that

the user just used, using a very simple test map. If it works, it’s not this.

4.  XSL Stylesheet. Here’s where it gets a bit more fun. Check the temporary directory on the outgen server. Is there a .fo file?

§ NO -> The XSL transformation failed. Potentially a stylesheet problem.

§ YES -> The XSL transformation succeeded.

Finally, the culprit

Content issue makes .fo invalid. Even more fun! Check the OT log. You will see something like:

AH error: 8205 (200D) 4 Transformed FO is invalid. §  This is actually quite common when a ditaval file excludes required sub-

elements. §  For example, conditioned table rows removed from a ditaval result in a

table with no rows = invalid.

Missing font. Check the OT log for something like this: AH: 1928 (0788) 1 Font file error: [filename] §  This is also relatively common and can occur when a new font is

introduced on a dev server and not promoted to production with the transformation scenario.

§  In fact there are actually many Antenna House errors that can prevent a PDF from being generated.

§  https://www.antennahouse.com/product/ahf60/docs/ahf-error.html

What If It’s Not Any of These Things?

• Float down the river… § Read all logs beginning at the start of the data flow.

Wrapper -> build html -> OT § Look for any abnormalities

•  If you don’t know what an error means, try googling it.

• Search IXIASOFT’s documentation •  If you’re still not sure, http://otrs.ixiasoft.com

Takeaways

LAWS UNDERSTAND LOGS

RIVER (FLOW)

Q&A

Any questions now? Speak up! Time permitting J Any questions later? [email protected] More info on troubleshooting? Maintaining a DITA CMS Deployment (Toolsmith’s Guide)

troubleshooting: the two laws - ixiasoft user conference 2016

Software