ChatGPT as a Documentation Tool

It comes as no surprise that most developers do not like writing documentation.  As a result, documentation is often pushed to the side and, more often than not, haphazardly put together.  This is a serious problem since written documentation is the primary way developers communicate how a software system should work and be utilized.  As such, a poorly documented system can render it useless and confine it to the cyber trash heap long before its time.  However, with the rise of new tools like ChatGPT, poor documentation may no longer be a problem.  This article is going to explore how ChatGPT can be leveraged as a code documentation tool and examine the results to see just how well it can document code.

What is quality documentation?

If you ask five developers what they consider to be good documentation, you’re probably going to get five different answers.  What’s considered good documentation will vary among languages, IDEs, organizational coding standards, and preferences.  As such, for this article, “good” documentation is going to have the following parameters:

1.     The documentation will provide context on how a block of code works or how to utilize it. 

2.     The documentation will be logical and consistent.  This means that the documentation will be formatted and worded logically to provide context across all code blocks that ChatGPT documents.

3.     The autogenerated code documentation will not clutter the code or distract the reader from the task at hand.

Again, it is important to remember that there are many different opinions on what quality documentation looks like, and the points above are just high-level metrics to hit.    

What is the documentation?

Documentation is a very broad term that can mean a lot of things.  It can mean simply the documentation that comes packaged with the codebases like JavaDocs, or it could be a whole ecosystem that ranges from low-level code comments to YouTube videos and Wikipedia articles.  In short, documentation can be anything that helps relay information from one developer to another.  As such, for this article, the term documentation is going to be limited to code comments and high-level documentation like JavaDocs.  Also, most languages have their own best practices for documentation, so for this article, ChatGPT will generate documentation for Java.  Though Java will be the example language, there should only be minor tweaks, if any, to generate documentation for any other language. 

JavaDocs

JavaDocs should provide high-level documentation to relay information about how to use a piece of code to a developer that does not have access to the codebase.  This means that key features like what the methods do, arguments, and return types should be noted.  With that in mind, let’s generate some JavaDocs for the following method,

public static void add(int a, int b) {
      System.out.println(a + b);
}

We’re going to generate the JavaDocs for this method using the following prompt:

Generate some javadocs for this method public static void add(int a, int b) { System.out.println(a + b); }

When the prompt is executed, it will produce the following output:

/**
 * Calculates the sum of two integers and prints the result.
 *
 * @param a The first integer operand.
 * @param b The second integer operand.
 */
public static void add(int a, int b) {
    System.out.println(a + b);
}

In this case, ChatGPT clearly defined what the arguments are, the datatypes of the arguments, and what the method is meant to do with straightforward prompts.  It is important to note that this method is straightforward, so the next thing to explore is a more complex method.

The next method to try to automatically document will be the following:

public static double divide(int a, int b) {
double div = 0;
      if (a > b) {
            div = a / b ;
      }else {
            div = b / a;
      }
      return div;
}

For this method, the following prompt will be run. 

Write only JavaDocs for this method public static double divide(int a, int b) { double div = 0; if (a > b) { div = a / b ; }else { div = b / a; } return div; }

After the prompt is run, we should be met with something similar to the following:

/**
 * Calculates the division of two numbers.
 *
 * @param a The numerator.
 * @param b The denominator.
 * @return The division result.
 */

These JavaDocs look very similar to the JavaDocs for the simple method.  They provide the arguments, return type, and the purpose of the method.   The one thing that was kind of ambiguous is that ChatGPT attempted to identify the variables as always the numerator or denominator.  From the code, we can see that this isn’t always true; however, if a developer is checking, they should be able to correct it fairly easily. 

Now, at first glance, this will appear as more trouble than it is worth.  A codebase of significant size is going to have many classes that are comprised of many different methods.  As such, copying and pasting methods one by one is going to be a herculean task.  However, ChatGPT can still be leveraged to speed up high-level code documentation.  Consider the following prompt:

Write JavaDocs for these methods public static void add(int a, int b) { System.out.println(a + b); } public static double divide(int a, int b) { double div = 0; if (a > b) { div = a / b ; }else { div = b / a; } return div; }

This prompt will produce the following results:

/**
 * Adds two numbers and prints the result.
 *
 * @param a The first number.
 * @param b The second number.
 */
 
/**
 * Calculates the division of two numbers and returns the result.
 *
 * @param a The numerator.
 * @param b The denominator.
 * @return The division result.
 */

Again, even with multiple methods, ChatGPT returned JavaDocs that are quality and provided enough information for a developer to use, with the exception of the same hiccup for the division method that was previously explored.  In all, ChatGPT did a decent job with writing JavaDocs.

Used in this manner, ChatGPT can greatly enhance the speed and quality of development.  When using ChatGPT, multiple methods can be pasted into ChatGPT, and all the necessary JavaDocs are created.  As anyone who has ever had to write this type of documentation knows, it can be very time-consuming and mistake-prone.  However, with ChatGPT, all one has to do is input the code and let the AI system do the rest.  Ultimately ChatGPT will free up developers for more pressing tasks, and the AI will produce quality JavaDocs without making rudimentary errors like forgetting to include a parameter or return type.  With that, another question is how well ChatGPT can be used to write code comments. 

Code Comments

In many cases, it is difficult for a 3^rd party to write effective code comments.  To provide the most context to the code, the entity that is writing the code should also write the comments.  As such, the first thing that needs to be explored is how ChatGPT writes code comments.  To do this, consider the following code snippet:

public static void add(int a, int b) {
      System.out.println(a + b);
}

This code is very basic; however, in a quality codebase, even a method this simple should be documented.  This method would probably take as much time to document as it would be to write; this means it is a perfect candidate for ChatGPT.  To have ChatGPT document the code snippet runs the following prompt:

Write a few comments to provide context for the following method public static void add(int a, int b) { System.out.println(a + b); }

After running this prompt, ChatGPT will produce the following:

public static void add(int a, int b) {
    // This method takes two integer parameters, 'a' and 'b'.
    // It adds the values of 'a' and 'b' together and prints the result.
    System.out.println(a + b);
    // The sum of 'a' and 'b' is calculated and printed to the console using System.out.println().
 
    // This method does not return any value (void).
}

At first glance, one may notice that ChatGPT generated several comments.  This can be a pro as much as it can be a con since it can bloat the codebase.  With that, one may notice that the output from ChatGPT does have a couple of unnecessary comments that can bloat the code.  For example, there is a comment that says that the method takes in two integers, and another says that the method returns void.  For code comments, those comments are not necessary as they don’t contribute to the overall context of the code.  However, if the prompt is modified to,

Write a comment that describes what this method does public static void add(int a, int b) { System.out.println(a + b); }

It will produce the following:

public static void add(int a, int b) {
    // This method calculates the sum of two integers, 'a' and 'b',
    // and displays the result by printing it to the console.
}

This prompt produced two comments that, when combined, form one complete sentence that describes what the method does. 

Now, this was for a relatively simple method.  As any programmer knows, a method will probably be larger and more complex than the add method.  As such, let’s consider the division method again:

public static double divide(int a, int b) {
      double div = 0;
      if (a > b) {
            div = a / b ;
      }else {
            div = b / a;
      }
      return div;
}

This method is a little more complex than the add method.  In this case, this method will return the quotient of a and b if a is greater than b or the quotient of b and a if b is greater than a.  With that, if the following prompt is inputted,

Write comments to add context to this method public static double divide(int a, int b) { double div = 0; if (a > b) { div = a / b ; }else { div = b / a; } return div; }

it will produce the following,

public static double divide(int a, int b) {
    // This method calculates the division of two integers and returns the result.
 
    double div = 0;
    // Variable to store the division result.
 
    if (a > b) {
        div = a / b;
        // If 'a' is greater than 'b', divide 'a' by 'b' and assign the result to 'div'.
    } else {
        div = b / a;
        // If 'b' is greater than or equal to 'a', divide 'b' by 'a' and assign the result to 'div'.
    }
 
    return div;
    // Return the division result.
}

In short, ChatGPT will document every line of code again.  The comments do provide code context, but the comments don’t provide anything that can’t simply be read in the code.  Also, if this prompt is modified, ChatGPT, at the time of writing this article, will still produce relatively useless comments.

Finally, if a prompt like the following is used:

Write only the necessary comments for a developer to use public static double divide(int a, int b) { double div = 0; if (a > b) { div = a / b ; }else { div = b / a; } return div; }

it will yield:

// Calculates the division of two numbers and returns the result.
// Parameters:
//   a: The numerator.
//   b: The denominator.
// Returns:
//   The division result.

Of all the queries explored thus far, this one arguably yielded the best results.  In this case, the comments are what was generated in the last queries, but the format is much cleaner and easier to modify.  This block could be easily pasted into the codebase, and the unnecessary lines deleted. 

In all, it took a very specific prompt to produce marginally acceptable code comments.  At best, the code comments that were produced didn’t provide any extra context that could not be deduced from reading the code.  It can also be said that the code comments that were produced bloated the code and, to some, may have made it more confusing.  As such, for code comments, ChatGPT probably isn’t the best tool to use.  At best, a developer will have to remove unnecessary lines of comments and probably have to re-write many of them as well. There is also the issue of having to produce a prompt that is specific enough to generate proper comments. 

In all, whether a person should use ChatGPT as a code comment generator is up to them.  In theory, the comment produced could be leveraged in places like education, where code examples need to be heavily commented on to provide context to those who may not have a background in the language.  However, in terms of production code, though it will ultimately depend on the organization’s coding standard, ChatGPT will not produce code comments that would be mergeable in many places. 

Keytake Aways  

In terms of codebase comments, ChatGPT is hit-and-miss.  As was seen, the code comments that ChatGPT produced were reminiscent of a college-level developer.  That is, ChatGPT commented on every line of code and only stated the obvious.  Since ChatGPT commented on every line of code, it can be argued that it bloated the codebase to a degree.  However, when a very specific prompt was run, it produced comments similar to what would be found in JavaDocs and what is expected by many organizations.  However, in terms of JavaDocs, ChatGPT shined.  The JavaDocs that ChatGPT produced were all very well written and provided the correct amount of information for a developer to easily digest and apply. 

As such, a few things can be summarized with what was explored.

1.     Queries have to be very specific when it comes to code comments.

2.     ChatGPT tends to produce unnecessary code comments that can bloat the codebase. 

3.     Depending on the type/quality of code comments, ChatGPT may not be the ideal tool for automatic code documentation.

4.     ChatGPT produces documentation akin to JavaDocs better than comments in the codebase.

Summary

In summary, what constitutes quality code documentation is often up to a team.  However, by many standards, ChatGPT tends to produce unnecessary code comments that don’t add much context and can easily bloat the codebase.  However, for higher-level documentation like JavaDocs, ChatGPT is an excellent tool that provides the proper amount of information.  In all, it probably isn’t the best idea to use ChatGPT as a means to generate comments for software written by a human, but it can be used to quickly produce higher-level documentation such as JavaDocs. As was seen, multiple methods can easily be documented in a matter of seconds using ChatGPT.  As such, in terms of productivity, when it comes to higher-level documentation, ChatGPT can be a great productivity tool that could help speed up development. 

Author Bio

M.T. White has been programming since the age of 12. His fascination with robotics flourished when he was a child programming microcontrollers such as Arduino. M.T. currently holds an undergraduate degree in mathematics, and a master's degree in software engineering, and is currently working on an MBA in IT project management. M.T. is currently working as a software developer for a major US defense contractor and is an adjunct CIS instructor at ECPI University. His background mostly stems from the automation industry where he programmed PLCs and HMIs for many different types of applications. M.T. has programmed many different brands of PLCs over the years and has developed HMIs using many different tools.

Author of the book: Mastering PLC Programming