The Incident came from Inside the House: Health Check Horror Stories series

Reading Time: 4 minutes

A Jira server update might usually end with a green checkmark. This one left the lights flickering.

This on-prem system was ticking along, receiving support from Highway Three’s team as needed. The client team had moved Jira to 11.3 and Confluence to 10.2. The plan was simple: apply the upgrades, run the checks, and get back to work.

Instead, the systems came back online with something following them home.

health check horror stories

When the house went quiet

Incoming-only OAuth2 application links stopped working. The doors between Jira, Confluence, and the internal applications they depended on had slammed shut. Atlassian Support had already tried three fixes before the week began. None of them worked.

Then Jira indexing failed.

Then Velocity email templates stopped behaving.

And in Confluence, older Roadmap Planner macros could still be viewed, but no one could edit them. A plugin had quietly disabled itself during the upgrade, like a room in the house disappearing overnight. Re-enabling Team Calendars gave the team a workaround, but by then the whole instance was creaking. šŸšļø

Suit up, Health Check Crew

Our consultant Rob found himself standing in the hallway with a flashlight. He became the incident lead: coordinating the response across two Sev-2 tickets, keeping the customer and Highway Three aligned through after-hours ā€œbounceā€ windows, and driving the collection of support bundles, logs, and database query results that Atlassian engineering needed.

The jira server update was complete - the haunting had begun

Then he started finding the practical fixes.

He isolated the specific custom field that was constantly breaking Jira indexing and restored indexing by having the field re-saved. He connected the Roadmap Planner editing problem to the Team Calendars plugin that had disabled itself during the upgrade, then found the workaround by re-enabling it. And he helped Atlassian identify the likely application-links data-model mismatch, giving Atlassian a concrete root-cause path instead of a collection of unrelated ghosts.

Rob also built Jira and Confluence sandbox environments so proposed database fixes could be tested safely before anyone touched production. The Confluence sandbox had its own authentication problems, because apparently the house had more than one haunted entrance. šŸ‘»

Something was living in the walls

The working theory was that Jira 11.3 and Confluence 10.2 had moved to a new, more cloud-architecture-friendly application-links data model. The problem appeared to be that the REST API was still looking for that data in the wrong place.

That mismatch kept the OAuth2 application-links issue broken even after Jira reached 11.3.8. The Jira server update had not just changed the house. It had changed the floor plan without updating the map. šŸ 

Atlassian opened two Sev-2 tickets, with engineering resources recreating the customer’s pre-upgrade configuration in a lab. At least one issue was confirmed as reproducible and new, with no existing fix. Several additional bugs were filed across Jira, Confluence, possibly Crowd, and multiple plugins that had once been Marketplace apps and were now part of the Atlassian system.

Not every horror story ends by Friday

By the end of the week, Rob had restored Jira indexing, found a workaround for Roadmap Planner, built safe testing environments, and handed Atlassian the evidence needed to continue the investigation. The team had a clearer picture of what had changed and a safer way to test the next proposed fix. ā›‘ļø

But the application-links issue was still unresolved. On July 24, everything was back in Atlassian Support’s hands, along with the logs, query results, and evidence needed for engineering to continue the investigation.

That is the part of upgrades no one puts in the checklist: sometimes the system comes back online, but the incident is only beginning. 😳

The final root cause

Three weeks after the initial Jira Server update, the root cause finally became clear. The teams compared Atlassian’s lab environments with the client’s sandboxes and production systems, which were still running Jira 11.3 and Confluence 10.2. That clue revealed what had actually happened.

The full explanation turned out to be more complicated than a single bug. It was a combination of three things:

  1. The Jira and Confluence LTS releases had advanced their code and architecture, changing how unidirectional, inbound-only OAuth2 link handshakes were handled and changing the broader application-links data model.
  2. Those changes exposed outdated custom client code in multiple internal applications. The trail eventually led to a third-party library inside that code.
  3. The issue was difficult for external partners like Highway Three to reproduce internally because we did not have access to the customer’s specific desktop applications. That application-link and OAuth path had also worked without issue for more than a decade of support, so there was no obvious warning that anything was waiting in the walls. šŸ¦‡

In other words, the updated Atlassian system code exposed a pre-existing problem in the customer’s custom applications. The upgrade was not the only culprit, and the old code was not breaking on its own. The two layers had to meet before the monster could show itself. 🧟 🪦

An upgrade is not a health check. It can expose old assumptions, disabled plugins, undocumented dependencies, and data models that no longer line up. The question is not whether the next release will have a known issue. It is whether you will have a map when something unexpected starts moving behind the walls.

What’s lurking in your instance? A health check will show you.

Discover more from Highway Three

Subscribe now to keep reading and get access to the full archive.

Continue reading