parsing documentation comments in clangllvm.org/devmtg/2012-11/gribenko_commentparsing.pdfparsing...
TRANSCRIPT
![Page 1: Parsing Documentation Comments in Clangllvm.org/devmtg/2012-11/Gribenko_CommentParsing.pdfParsing Documentation Comments in Clang DmitriGribenkohgribozavr@gmail.comi HighPerformanceComputingCenteratNTUU¾KPI¿](https://reader030.vdocuments.mx/reader030/viewer/2022040508/5e4d5e5ac5f24a0b19608872/html5/thumbnails/1.jpg)
Parsing Documentation Comments in Clang
Dmitri Gribenko 〈[email protected]〉High Performance Computing Center at NTUU «KPI»
LLVM Developers’ MeetingNovember 8, 2012
![Page 2: Parsing Documentation Comments in Clangllvm.org/devmtg/2012-11/Gribenko_CommentParsing.pdfParsing Documentation Comments in Clang DmitriGribenkohgribozavr@gmail.comi HighPerformanceComputingCenteratNTUU¾KPI¿](https://reader030.vdocuments.mx/reader030/viewer/2022040508/5e4d5e5ac5f24a0b19608872/html5/thumbnails/2.jpg)
Outline
• Motivation• Implementation details• User-visible features in Clang and libclang
![Page 3: Parsing Documentation Comments in Clangllvm.org/devmtg/2012-11/Gribenko_CommentParsing.pdfParsing Documentation Comments in Clang DmitriGribenkohgribozavr@gmail.comi HighPerformanceComputingCenteratNTUU¾KPI¿](https://reader030.vdocuments.mx/reader030/viewer/2022040508/5e4d5e5ac5f24a0b19608872/html5/thumbnails/3.jpg)
Clang is (was) ignoring documentation comments
• Comments tend to bitrot.• Syntax errors in markup are not caught.• Documentation extraction tools don’t really understand C++.• Tools based on Clang libraries will also ignore comments:
• IDE source editor tools;• refactoring tools.
![Page 4: Parsing Documentation Comments in Clangllvm.org/devmtg/2012-11/Gribenko_CommentParsing.pdfParsing Documentation Comments in Clang DmitriGribenkohgribozavr@gmail.comi HighPerformanceComputingCenteratNTUU¾KPI¿](https://reader030.vdocuments.mx/reader030/viewer/2022040508/5e4d5e5ac5f24a0b19608872/html5/thumbnails/4.jpg)
Documentation support: goals
• Help to keep comments up to date and syntactically correct.• Enhance Clang-based tools with comments support:
• IDE source editor tools;• refactoring tools.
• (Maybe) Build a better documentation extraction tool.
![Page 5: Parsing Documentation Comments in Clangllvm.org/devmtg/2012-11/Gribenko_CommentParsing.pdfParsing Documentation Comments in Clang DmitriGribenkohgribozavr@gmail.comi HighPerformanceComputingCenteratNTUU¾KPI¿](https://reader030.vdocuments.mx/reader030/viewer/2022040508/5e4d5e5ac5f24a0b19608872/html5/thumbnails/5.jpg)
Documentation comments≈
Doxygen
![Page 6: Parsing Documentation Comments in Clangllvm.org/devmtg/2012-11/Gribenko_CommentParsing.pdfParsing Documentation Comments in Clang DmitriGribenkohgribozavr@gmail.comi HighPerformanceComputingCenteratNTUU¾KPI¿](https://reader030.vdocuments.mx/reader030/viewer/2022040508/5e4d5e5ac5f24a0b19608872/html5/thumbnails/6.jpg)
Implementation
• Finding comments.• Attaching comments to declarations.• Comment parsing.• Comment AST.
![Page 7: Parsing Documentation Comments in Clangllvm.org/devmtg/2012-11/Gribenko_CommentParsing.pdfParsing Documentation Comments in Clang DmitriGribenkohgribozavr@gmail.comi HighPerformanceComputingCenteratNTUU¾KPI¿](https://reader030.vdocuments.mx/reader030/viewer/2022040508/5e4d5e5ac5f24a0b19608872/html5/thumbnails/7.jpg)
Attaching comments to AST nodes
• Teach Clang parser about tok::comment:• would complicate the parser;• would add lots of checks for tok::comment;• no easy way to turn off.
• Attach comments after the AST is built.
![Page 8: Parsing Documentation Comments in Clangllvm.org/devmtg/2012-11/Gribenko_CommentParsing.pdfParsing Documentation Comments in Clang DmitriGribenkohgribozavr@gmail.comi HighPerformanceComputingCenteratNTUU¾KPI¿](https://reader030.vdocuments.mx/reader030/viewer/2022040508/5e4d5e5ac5f24a0b19608872/html5/thumbnails/8.jpg)
Finding raw comments
![Page 9: Parsing Documentation Comments in Clangllvm.org/devmtg/2012-11/Gribenko_CommentParsing.pdfParsing Documentation Comments in Clang DmitriGribenkohgribozavr@gmail.comi HighPerformanceComputingCenteratNTUU¾KPI¿](https://reader030.vdocuments.mx/reader030/viewer/2022040508/5e4d5e5ac5f24a0b19608872/html5/thumbnails/9.jpg)
Documentation comments
/// commentvoid f(); int a; ///< comment
/** comment */void g(); int b; /**< comment */
/*! comment */void h(); int c; /*!< comment */
![Page 10: Parsing Documentation Comments in Clangllvm.org/devmtg/2012-11/Gribenko_CommentParsing.pdfParsing Documentation Comments in Clang DmitriGribenkohgribozavr@gmail.comi HighPerformanceComputingCenteratNTUU¾KPI¿](https://reader030.vdocuments.mx/reader030/viewer/2022040508/5e4d5e5ac5f24a0b19608872/html5/thumbnails/10.jpg)
"Almost documentation" comments
struct a {int x; //< commentint y; /*< comment */
};
![Page 11: Parsing Documentation Comments in Clangllvm.org/devmtg/2012-11/Gribenko_CommentParsing.pdfParsing Documentation Comments in Clang DmitriGribenkohgribozavr@gmail.comi HighPerformanceComputingCenteratNTUU¾KPI¿](https://reader030.vdocuments.mx/reader030/viewer/2022040508/5e4d5e5ac5f24a0b19608872/html5/thumbnails/11.jpg)
"Almost documentation" comments
struct a {int x; // /< commentint y; /* *< comment */
};
example.c:8:10: warning: not a Doxygen trailing commentint x; //< comment
∧~~///<
![Page 12: Parsing Documentation Comments in Clangllvm.org/devmtg/2012-11/Gribenko_CommentParsing.pdfParsing Documentation Comments in Clang DmitriGribenkohgribozavr@gmail.comi HighPerformanceComputingCenteratNTUU¾KPI¿](https://reader030.vdocuments.mx/reader030/viewer/2022040508/5e4d5e5ac5f24a0b19608872/html5/thumbnails/12.jpg)
Attaching comments to AST nodes
![Page 13: Parsing Documentation Comments in Clangllvm.org/devmtg/2012-11/Gribenko_CommentParsing.pdfParsing Documentation Comments in Clang DmitriGribenkohgribozavr@gmail.comi HighPerformanceComputingCenteratNTUU¾KPI¿](https://reader030.vdocuments.mx/reader030/viewer/2022040508/5e4d5e5ac5f24a0b19608872/html5/thumbnails/13.jpg)
Attaching comments to AST nodes
![Page 14: Parsing Documentation Comments in Clangllvm.org/devmtg/2012-11/Gribenko_CommentParsing.pdfParsing Documentation Comments in Clang DmitriGribenkohgribozavr@gmail.comi HighPerformanceComputingCenteratNTUU¾KPI¿](https://reader030.vdocuments.mx/reader030/viewer/2022040508/5e4d5e5ac5f24a0b19608872/html5/thumbnails/14.jpg)
Attaching comments to AST nodes
![Page 15: Parsing Documentation Comments in Clangllvm.org/devmtg/2012-11/Gribenko_CommentParsing.pdfParsing Documentation Comments in Clang DmitriGribenkohgribozavr@gmail.comi HighPerformanceComputingCenteratNTUU¾KPI¿](https://reader030.vdocuments.mx/reader030/viewer/2022040508/5e4d5e5ac5f24a0b19608872/html5/thumbnails/15.jpg)
Attaching comments, eager mode
![Page 16: Parsing Documentation Comments in Clangllvm.org/devmtg/2012-11/Gribenko_CommentParsing.pdfParsing Documentation Comments in Clang DmitriGribenkohgribozavr@gmail.comi HighPerformanceComputingCenteratNTUU¾KPI¿](https://reader030.vdocuments.mx/reader030/viewer/2022040508/5e4d5e5ac5f24a0b19608872/html5/thumbnails/16.jpg)
Comments from "related" declarations
/// \brief Does foo./// \param a blah.void f(int a);
void f(int b) { // picks up the comment above.}
![Page 17: Parsing Documentation Comments in Clangllvm.org/devmtg/2012-11/Gribenko_CommentParsing.pdfParsing Documentation Comments in Clang DmitriGribenkohgribozavr@gmail.comi HighPerformanceComputingCenteratNTUU¾KPI¿](https://reader030.vdocuments.mx/reader030/viewer/2022040508/5e4d5e5ac5f24a0b19608872/html5/thumbnails/17.jpg)
Comments from "related" declarations
struct A {/// \brief Does foo.virtual void f();
};
struct B : public A {virtual void f(); // picks up A::f()’s comment.
};
![Page 18: Parsing Documentation Comments in Clangllvm.org/devmtg/2012-11/Gribenko_CommentParsing.pdfParsing Documentation Comments in Clang DmitriGribenkohgribozavr@gmail.comi HighPerformanceComputingCenteratNTUU¾KPI¿](https://reader030.vdocuments.mx/reader030/viewer/2022040508/5e4d5e5ac5f24a0b19608872/html5/thumbnails/18.jpg)
Comment parsing
![Page 19: Parsing Documentation Comments in Clangllvm.org/devmtg/2012-11/Gribenko_CommentParsing.pdfParsing Documentation Comments in Clang DmitriGribenkohgribozavr@gmail.comi HighPerformanceComputingCenteratNTUU¾KPI¿](https://reader030.vdocuments.mx/reader030/viewer/2022040508/5e4d5e5ac5f24a0b19608872/html5/thumbnails/19.jpg)
Comment AST
• Root FullComment node has children which are block content.• ParagraphComment nodes have children which are inlinecontent.
• Block content:• paragraph;• block command (e. g., \brief or \verbatim).
• Inline content:• text;• inline command (e. g., \c);• HTML tag.
![Page 20: Parsing Documentation Comments in Clangllvm.org/devmtg/2012-11/Gribenko_CommentParsing.pdfParsing Documentation Comments in Clang DmitriGribenkohgribozavr@gmail.comi HighPerformanceComputingCenteratNTUU¾KPI¿](https://reader030.vdocuments.mx/reader030/viewer/2022040508/5e4d5e5ac5f24a0b19608872/html5/thumbnails/20.jpg)
Comment AST
• Root FullComment node has children which are block content.• ParagraphComment nodes have children which are inlinecontent.
• Block content:• paragraph;• block command (e. g., \brief or \verbatim).
• Inline content:• text;• inline command (e. g., \c);• HTML tag.
![Page 21: Parsing Documentation Comments in Clangllvm.org/devmtg/2012-11/Gribenko_CommentParsing.pdfParsing Documentation Comments in Clang DmitriGribenkohgribozavr@gmail.comi HighPerformanceComputingCenteratNTUU¾KPI¿](https://reader030.vdocuments.mx/reader030/viewer/2022040508/5e4d5e5ac5f24a0b19608872/html5/thumbnails/21.jpg)
Comment AST
/// \brief Does something with \p str./// \param [in] Str the string./// \returns a modified string.void do_something(const std::string &str);
![Page 22: Parsing Documentation Comments in Clangllvm.org/devmtg/2012-11/Gribenko_CommentParsing.pdfParsing Documentation Comments in Clang DmitriGribenkohgribozavr@gmail.comi HighPerformanceComputingCenteratNTUU¾KPI¿](https://reader030.vdocuments.mx/reader030/viewer/2022040508/5e4d5e5ac5f24a0b19608872/html5/thumbnails/22.jpg)
User-visible features
• Diagnostics for comments.• Enhanced code completion APIs (libclang).• Export comments as XML (libclang).• Performance.
![Page 23: Parsing Documentation Comments in Clangllvm.org/devmtg/2012-11/Gribenko_CommentParsing.pdfParsing Documentation Comments in Clang DmitriGribenkohgribozavr@gmail.comi HighPerformanceComputingCenteratNTUU¾KPI¿](https://reader030.vdocuments.mx/reader030/viewer/2022040508/5e4d5e5ac5f24a0b19608872/html5/thumbnails/23.jpg)
Diagnostics
/// \brief Does something with \p str./// \param [in] Str the string./// \returns a modified string.void do_something(const std::string &str);
![Page 24: Parsing Documentation Comments in Clangllvm.org/devmtg/2012-11/Gribenko_CommentParsing.pdfParsing Documentation Comments in Clang DmitriGribenkohgribozavr@gmail.comi HighPerformanceComputingCenteratNTUU¾KPI¿](https://reader030.vdocuments.mx/reader030/viewer/2022040508/5e4d5e5ac5f24a0b19608872/html5/thumbnails/24.jpg)
Diagnostics
/// \brief Does something with \p str./// \param [in] Str the string./// \returns a modified string.void do_something(const std::string &str);
example.cc:4:17: warning: parameter ’Str’ not foundin the function declaration [-Wdocumentation]
/// \param [in] Str the string.∧~~
example.cc:4:17: note: did you mean ’str’?/// \param [in] Str the string.
∧~~str
![Page 25: Parsing Documentation Comments in Clangllvm.org/devmtg/2012-11/Gribenko_CommentParsing.pdfParsing Documentation Comments in Clang DmitriGribenkohgribozavr@gmail.comi HighPerformanceComputingCenteratNTUU¾KPI¿](https://reader030.vdocuments.mx/reader030/viewer/2022040508/5e4d5e5ac5f24a0b19608872/html5/thumbnails/25.jpg)
Diagnostics
/// \brief Does something with \p str./// \param [in] Str the string./// \returns a modified string.void do_something(const std::string &str);
example.cc:5:6: warning: ’\returns’ command usedin a comment that is attached to a functionreturning void [-Wdocumentation]
/// \returns a modified string.~∧~~~~~~~~~~~~~~~~~~~~~~~~~
![Page 26: Parsing Documentation Comments in Clangllvm.org/devmtg/2012-11/Gribenko_CommentParsing.pdfParsing Documentation Comments in Clang DmitriGribenkohgribozavr@gmail.comi HighPerformanceComputingCenteratNTUU¾KPI¿](https://reader030.vdocuments.mx/reader030/viewer/2022040508/5e4d5e5ac5f24a0b19608872/html5/thumbnails/26.jpg)
Diagnostics
/// \param x value of X coordinate./// \param x value of Y coordinate.void do_something(int x, int y);
example.cc:2:12: warning: parameter ’x’ is alreadydocumented [-Wdocumentation]
/// \param x value of Y coordinate.∧
![Page 27: Parsing Documentation Comments in Clangllvm.org/devmtg/2012-11/Gribenko_CommentParsing.pdfParsing Documentation Comments in Clang DmitriGribenkohgribozavr@gmail.comi HighPerformanceComputingCenteratNTUU¾KPI¿](https://reader030.vdocuments.mx/reader030/viewer/2022040508/5e4d5e5ac5f24a0b19608872/html5/thumbnails/27.jpg)
Diagnostics
/// \brief Does something./// \deprecatedvoid do_something();
example.cc:4:6: warning: declaration is marked with’\deprecated’ command but does not havea deprecation attribute
/// \deprecated~∧~~~~~~~~~
example.cc:5:19: note: add a deprecation attributeto the declaration to silence this warning
void do_something();∧
__attribute__((deprecated))
![Page 28: Parsing Documentation Comments in Clangllvm.org/devmtg/2012-11/Gribenko_CommentParsing.pdfParsing Documentation Comments in Clang DmitriGribenkohgribozavr@gmail.comi HighPerformanceComputingCenteratNTUU¾KPI¿](https://reader030.vdocuments.mx/reader030/viewer/2022040508/5e4d5e5ac5f24a0b19608872/html5/thumbnails/28.jpg)
Diagnostics
/// \brief Does something./// \deprecatedvoid do_something();
example.cc:4:6: warning: declaration is marked with’\deprecated’ command but does not havea deprecation attribute
/// \deprecated~∧~~~~~~~~~
example.cc:5:19: note: add a deprecation attributeto the declaration to silence this warning
void do_something();∧
__attribute__((deprecated))
![Page 29: Parsing Documentation Comments in Clangllvm.org/devmtg/2012-11/Gribenko_CommentParsing.pdfParsing Documentation Comments in Clang DmitriGribenkohgribozavr@gmail.comi HighPerformanceComputingCenteratNTUU¾KPI¿](https://reader030.vdocuments.mx/reader030/viewer/2022040508/5e4d5e5ac5f24a0b19608872/html5/thumbnails/29.jpg)
Diagnostics
#define MY_DEPRECATED __attribute__((deprecated))/// \brief Does something./// \deprecatedvoid do_something();
example.cc:4:6: warning: declaration is marked with’\deprecated’ command but does not havea deprecation attribute
/// \deprecated~∧~~~~~~~~~
example.cc:5:19: note: add a deprecation attributeto the declaration to silence this warning
void do_something();∧
MY_DEPRECATED
![Page 30: Parsing Documentation Comments in Clangllvm.org/devmtg/2012-11/Gribenko_CommentParsing.pdfParsing Documentation Comments in Clang DmitriGribenkohgribozavr@gmail.comi HighPerformanceComputingCenteratNTUU¾KPI¿](https://reader030.vdocuments.mx/reader030/viewer/2022040508/5e4d5e5ac5f24a0b19608872/html5/thumbnails/30.jpg)
LLVM and Clang are 99% -Wdocumentation clean.
Thanks to James Dennett!
Please configure LLVM and Clang with"--with-extra-options=-Wdocumentation-Wno-documentation-deprecated-sync"
and help to clean up the last bits.
![Page 31: Parsing Documentation Comments in Clangllvm.org/devmtg/2012-11/Gribenko_CommentParsing.pdfParsing Documentation Comments in Clang DmitriGribenkohgribozavr@gmail.comi HighPerformanceComputingCenteratNTUU¾KPI¿](https://reader030.vdocuments.mx/reader030/viewer/2022040508/5e4d5e5ac5f24a0b19608872/html5/thumbnails/31.jpg)
LLVM and Clang are 99% -Wdocumentation clean.
Thanks to James Dennett!
Please configure LLVM and Clang with"--with-extra-options=-Wdocumentation-Wno-documentation-deprecated-sync"
and help to clean up the last bits.
![Page 32: Parsing Documentation Comments in Clangllvm.org/devmtg/2012-11/Gribenko_CommentParsing.pdfParsing Documentation Comments in Clang DmitriGribenkohgribozavr@gmail.comi HighPerformanceComputingCenteratNTUU¾KPI¿](https://reader030.vdocuments.mx/reader030/viewer/2022040508/5e4d5e5ac5f24a0b19608872/html5/thumbnails/32.jpg)
Enhanced code completion
Code completion provides:• text or patterns to insert;• priority;• availability information;
• brief documentation.
![Page 33: Parsing Documentation Comments in Clangllvm.org/devmtg/2012-11/Gribenko_CommentParsing.pdfParsing Documentation Comments in Clang DmitriGribenkohgribozavr@gmail.comi HighPerformanceComputingCenteratNTUU¾KPI¿](https://reader030.vdocuments.mx/reader030/viewer/2022040508/5e4d5e5ac5f24a0b19608872/html5/thumbnails/33.jpg)
Enhanced code completion
Code completion provides:• text or patterns to insert;• priority;• availability information;• brief documentation.
![Page 34: Parsing Documentation Comments in Clangllvm.org/devmtg/2012-11/Gribenko_CommentParsing.pdfParsing Documentation Comments in Clang DmitriGribenkohgribozavr@gmail.comi HighPerformanceComputingCenteratNTUU¾KPI¿](https://reader030.vdocuments.mx/reader030/viewer/2022040508/5e4d5e5ac5f24a0b19608872/html5/thumbnails/34.jpg)
XML export
• A comment can be converted to XML.• XML schema: bindings/xml/comment-xml-schema.rng.• XML document includes:
• parsed comment;• information about the declaration.
![Page 35: Parsing Documentation Comments in Clangllvm.org/devmtg/2012-11/Gribenko_CommentParsing.pdfParsing Documentation Comments in Clang DmitriGribenkohgribozavr@gmail.comi HighPerformanceComputingCenteratNTUU¾KPI¿](https://reader030.vdocuments.mx/reader030/viewer/2022040508/5e4d5e5ac5f24a0b19608872/html5/thumbnails/35.jpg)
XML export example
/// \brief Does quux./// \tparam T foo./// \param x bar.template<typename T> void foo(int x, T t);
<?xml version="1.0"?><Function templateKind="template" file="example.cc"
line="5" column="6"><Name>foo</Name><USR>c:@FT@>1#Tfoo#I#t0.0#</USR><Declaration>template <typename T>
void foo(int x, T t)</Declaration><Abstract><Para> Does quux. </Para></Abstract><TemplateParameters> ...<Parameters> ...
![Page 36: Parsing Documentation Comments in Clangllvm.org/devmtg/2012-11/Gribenko_CommentParsing.pdfParsing Documentation Comments in Clang DmitriGribenkohgribozavr@gmail.comi HighPerformanceComputingCenteratNTUU¾KPI¿](https://reader030.vdocuments.mx/reader030/viewer/2022040508/5e4d5e5ac5f24a0b19608872/html5/thumbnails/36.jpg)
Performance
• Parse comments only if -Wdocumentation is passed.• If you have lots (> 10000) of documentation comments, youmight see a < 5% regression in -fsyntax-only time.
• Parsing time is only a small fraction of compile time.• Comments from system headers are skipped during normalcompilation.
• There is no sense in parsing them for diagnostics.• Pass -fretain-comments-from-system-headers if you
want them back.
![Page 37: Parsing Documentation Comments in Clangllvm.org/devmtg/2012-11/Gribenko_CommentParsing.pdfParsing Documentation Comments in Clang DmitriGribenkohgribozavr@gmail.comi HighPerformanceComputingCenteratNTUU¾KPI¿](https://reader030.vdocuments.mx/reader030/viewer/2022040508/5e4d5e5ac5f24a0b19608872/html5/thumbnails/37.jpg)
DemoVim with clang_complete
![Page 38: Parsing Documentation Comments in Clangllvm.org/devmtg/2012-11/Gribenko_CommentParsing.pdfParsing Documentation Comments in Clang DmitriGribenkohgribozavr@gmail.comi HighPerformanceComputingCenteratNTUU¾KPI¿](https://reader030.vdocuments.mx/reader030/viewer/2022040508/5e4d5e5ac5f24a0b19608872/html5/thumbnails/38.jpg)
clang_complete with brief documentation support
https://github.com/gribozavr/clang_complete
![Page 39: Parsing Documentation Comments in Clangllvm.org/devmtg/2012-11/Gribenko_CommentParsing.pdfParsing Documentation Comments in Clang DmitriGribenkohgribozavr@gmail.comi HighPerformanceComputingCenteratNTUU¾KPI¿](https://reader030.vdocuments.mx/reader030/viewer/2022040508/5e4d5e5ac5f24a0b19608872/html5/thumbnails/39.jpg)
Future work
• Resolve parameter names in \p.• Resolve declaration references.• Attach comments to macros.• AST matchers?• Integrate this feature with other IDEs (maybe after clangd?)• Write a better documentation extraction tool.
• Fix the declaration pretty-printer.
![Page 40: Parsing Documentation Comments in Clangllvm.org/devmtg/2012-11/Gribenko_CommentParsing.pdfParsing Documentation Comments in Clang DmitriGribenkohgribozavr@gmail.comi HighPerformanceComputingCenteratNTUU¾KPI¿](https://reader030.vdocuments.mx/reader030/viewer/2022040508/5e4d5e5ac5f24a0b19608872/html5/thumbnails/40.jpg)
Summary
Now Clang can:• parse documentation comments;• find semantic errors in comments (try -Wdocumentation);• export comments as XML.
This work enables more awesome features in future!
![Page 41: Parsing Documentation Comments in Clangllvm.org/devmtg/2012-11/Gribenko_CommentParsing.pdfParsing Documentation Comments in Clang DmitriGribenkohgribozavr@gmail.comi HighPerformanceComputingCenteratNTUU¾KPI¿](https://reader030.vdocuments.mx/reader030/viewer/2022040508/5e4d5e5ac5f24a0b19608872/html5/thumbnails/41.jpg)
Q&A