Is Atlassian’s decision to forbid benchmarking potentially masking its degrading performance?

Last week, Atlassian software company released their updated ‘Atlassian Software License Agreement’ and their ‘Cloud Terms of Service’, which would be effective from the 1st of November, 2018.

Just as any general agreement, this one too mentions the scope, the users authorized, use of software, and so on. However, it has set up certain restrictions based on the performance of its software. As per the new agreement, benchmarking of Atlassian software is forbidden.

Restrictions on benchmarking

As per the discussion on the Atlassian Developer Community, Andy Broker highlighted two clauses from the restriction section have been highlighted, which includes:

(i) publicly disseminate information regarding the performance of the Software

Andy Broker, a marketplace vendor explains this clause as, “This sounds very much like the nonsense clause that Intel were derided for, regarding the performance of their CPU’s. Intel backtracked after being lambasted by the world, I can’t really understand how these points got into new Atlassian terms, surely the terms have had a technical review?

Just… why, given all the DC testing being done ongoing, this is an area where data we gathered may be interesting to prospective customers.”

(j) encourage or assist any third party to do any of the foregoing.
Andy Broker further adds, “So, we can’t guide/help a customer understand how to even measure performance to determine if they have a performance issue in the “Software”, e.g. generating a performance baseline before an ‘app’ is involved?

The result of this would appear sub-optimal for Customer and Vendors alike, the “Software” performance just becomes 3rd party App performance that we cannot ‘explain’ or ‘show’ to customers.”

Why Atlassian decided to forbid benchmarking?

As per a discussion thread on Hacker News, many users have stated their views on why Atlassian planned to forbid benchmarking on its software. According to a comment, “If the company bans benchmarking, then the product is slow.”

A user also stated that Atlassian software is slow, has annoying UX, and it is very inconsistent. This may be because most of its software is built using JIRA which has a Java backend and is not Node based. Jira cannot be rebuilt from scratch only slowly abstracted and broken up into smaller pieces. Also, around 3 years ago Atlassian forked behind the firewall for two distinct products, multi-tenant cloud and traditional to get into the cloud sector with a view to attracting more potential customers, thus increasing growth.

A user also stated, “Cloud was full buy into AWS, taking a behind the firewall product and making it multi-tenanted cloud-based is a huge job. A monolith service now becomes highly distributed so latency's obviously mounted up due to the many services to service interactions.”

The user further added, “some things which are heavily multi-threaded and built in statically compiled languages had to be built in single threaded Node.js because everyone is using Node.js and your language is now banned. It's not surprising there are noticeable performance differences.”

Another user has suggested, “the better way for a company to handle this concern is to proactively run and release benchmarks including commentary on the results, together with everything necessary for anyone to reproduce their results.” The user added that they can even fund a trustworthy neutral third party to perform benchmarking with proper disclosure of the funding.

To read the entire discussion in detail, head over to Hacker News.

Atlassian acquires OpsGenie, launches Jira Ops to make incident response more powerful

Atlassian sells Hipchat IP to Slack

Atlassian open sources Escalator, a Kubernetes autoscaler project