Clang-Doc hits an assertion in Sema when run on LLVM

[ilovepi]

When using clang-doc to generate documentation for LLVM, I'm seeing an assertion failure in Sema.

[1476/5643] Processing file /usr/local/google/home/paulkirth/llvm-project/llvm/lib/Support/FileCollector.cpp
clang-doc: /usr/local/google/home/paulkirth/llvm-project/clang/lib/Sema/SemaExprMember.cpp:993: ExprResult clang::Sema::BuildMemberReferenceExpr(Expr *, QualType, SourceLocation, bool, const CXXScopeSpec &, SourceLocation, NamedDecl *, LookupResult &, const TemplateArgumentListInfo *, const Scope *, bool, ActOnMemberAccessExtraArgs *): Assertion `!SS.isInvalid() && "nested-name-specifier cannot be invalid"' failed.
1 error generated.
 #0 0x000056425e24eb88 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (bin/clang-doc+0xe43b88)
 #1 0x000056425e24c85c llvm::sys::RunSignalHandlers() (bin/clang-doc+0xe4185c)
 #2 0x000056425e24f240 SignalHandler(int) Signals.cpp:0:0
 #3 0x00007fa2dafd0510 (/lib/x86_64-linux-gnu/libc.so.6+0x3c510)
 #4 0x00007fa2db01e16c __pthread_kill_implementation ./nptl/pthread_kill.c:44:76
 #5 0x00007fa2dafd0472 raise ./signal/../sysdeps/posix/raise.c:27:6
 #6 0x00007fa2dafba4b2 abort ./stdlib/abort.c:81:7
 #7 0x00007fa2dafba3d5 _nl_load_domain ./intl/loadmsgcat.c:1177:9
 #8 0x00007fa2dafc93a2 (/lib/x86_64-linux-gnu/libc.so.6+0x353a2)
 #9 0x000056425f538966 clang::Sema::BuildMemberReferenceExpr(clang::Expr*, clang::QualType, clang::SourceLocation, bool, clang::CXXScopeSpec const&, clang::SourceLocation, clang::NamedDecl*, clang::LookupResult&, clang::TemplateArgumentListInfo const*, clang::Scope const*, bool, clang::Sema::ActOnMemberAccessExtraArgs*) (bin/clang-doc+0x212d966)
#10 0x000056425f535aa1 clang::Sema::BuildMemberReferenceExpr(clang::Expr*, clang::QualType, clang::SourceLocation, bool, clang::CXXScopeSpec&, clang::SourceLocation, clang::NamedDecl*, clang::DeclarationNameInfo const&, clang::TemplateArgumentListInfo const*, clang::Scope const*, clang::Sema::ActOnMemberAccessExtraArgs*) (bin/clang-doc+0x212aaa1)
#11 0x000056425f539843 clang::Sema::ActOnMemberAccessExpr(clang::Scope*, clang::Expr*, clang::SourceLocation, clang::tok::TokenKind, clang::CXXScopeSpec&, clang::SourceLocation, clang::UnqualifiedId&, clang::Decl*) (bin/clang-doc+0x212e843)
#12 0x000056425ec061da clang::Parser::ParsePostfixExpressionSuffix(clang::ActionResult<clang::Expr*, true>) (bin/clang-doc+0x17fb1da)
#13 0x000056425ec08409 clang::Parser::ParseCastExpression(clang::Parser::CastParseKind, bool, bool&, clang::Parser::TypeCastState, bool, bool*) (bin/clang-doc+0x17fd409)
#14 0x000056425ec02dec clang::Parser::ParseAssignmentExpression(clang::Parser::TypeCastState) (bin/clang-doc+0x17f7dec)
#15 0x000056425ec02d19 clang::Parser::ParseExpression(clang::Parser::TypeCastState) (bin/clang-doc+0x17f7d19)
#16 0x000056425eccc911 clang::Parser::ParseReturnStatement() (bin/clang-doc+0x18c1911)
#17 0x000056425ecc5251 clang::Parser::ParseStatementOrDeclarationAfterAttributes(llvm::SmallVector<clang::Stmt*, 32u>&, clang::Parser::ParsedStmtContext, clang::SourceLocation*, clang::ParsedAttributes&, clang::ParsedAttributes&) (bin/clang-doc+0x18ba251)
#18 0x000056425ecc36d2 clang::Parser::ParseStatementOrDeclaration(llvm::SmallVector<clang::Stmt*, 32u>&, clang::Parser::ParsedStmtContext, clang::SourceLocation*) (bin/clang-doc+0x18b86d2)
#19 0x000056425eccf451 clang::Parser::ParseCompoundStatementBody(bool) (bin/clang-doc+0x18c4451)
#20 0x000056425ecd0732 clang::Parser::ParseFunctionStatementBody(clang::Decl*, clang::Parser::ParseScope&) (bin/clang-doc+0x18c5732)
#21 0x000056425ecfa080 clang::Parser::ParseLexedMethodDef(clang::Parser::LexedMethod&) (bin/clang-doc+0x18ef080)
#22 0x000056425ecf863a clang::Parser::ParseLexedMethodDefs(clang::Parser::ParsingClass&) (bin/clang-doc+0x18ed63a)
#23 0x000056425ec3a7a7 clang::Parser::ParseCXXMemberSpecification(clang::SourceLocation, clang::SourceLocation, clang::ParsedAttributes&, unsigned int, clang::Decl*) (bin/clang-doc+0x182f7a7)
#24 0x000056425ec378e7 clang::Parser::ParseClassSpecifier(clang::tok::TokenKind, clang::SourceLocation, clang::DeclSpec&, clang::Parser::ParsedTemplateInfo const&, clang::AccessSpecifier, bool, clang::Parser::DeclSpecContext, clang::ParsedAttributes&) (bin/clang-doc+0x182c8e7)
#25 0x000056425ec639cb clang::Parser::ParseDeclarationSpecifiers(clang::DeclSpec&, clang::Parser::ParsedTemplateInfo const&, clang::AccessSpecifier, clang::Parser::DeclSpecContext, clang::Parser::LateParsedAttrList*, clang::ImplicitTypenameContext) (bin/clang-doc+0x18589cb)
#26 0x000056425ebdfb1c clang::Parser::ParseDeclOrFunctionDefInternal(clang::ParsedAttributes&, clang::ParsedAttributes&, clang::ParsingDeclSpec&, clang::AccessSpecifier) (bin/clang-doc+0x17d4b1c)
#27 0x000056425ebdf78b clang::Parser::ParseDeclarationOrFunctionDefinition(clang::ParsedAttributes&, clang::ParsedAttributes&, clang::ParsingDeclSpec*, clang::AccessSpecifier) (bin/clang-doc+0x17d478b)
#28 0x000056425ebde5a9 clang::Parser::ParseExternalDeclaration(clang::ParsedAttributes&, clang::ParsedAttributes&, clang::ParsingDeclSpec*) (bin/clang-doc+0x17d35a9)
#29 0x000056425ec2cbff clang::Parser::ParseInnerNamespace(llvm::SmallVector<clang::Parser::InnerNamespaceInfo, 4u> const&, unsigned int, clang::SourceLocation&, clang::ParsedAttributes&, clang::BalancedDelimiterTracker&) (bin/clang-doc+0x1821bff)
#30 0x000056425ec2beb7 clang::Parser::ParseNamespace(clang::DeclaratorContext, clang::SourceLocation&, clang::SourceLocation) (bin/clang-doc+0x1820eb7)
#31 0x000056425ec5c9af clang::Parser::ParseDeclaration(clang::DeclaratorContext, clang::SourceLocation&, clang::ParsedAttributes&, clang::ParsedAttributes&, clang::SourceLocation*) (bin/clang-doc+0x18519af)
#32 0x000056425ebde0fd clang::Parser::ParseExternalDeclaration(clang::ParsedAttributes&, clang::ParsedAttributes&, clang::ParsingDeclSpec*) (bin/clang-doc+0x17d30fd)
#33 0x000056425ebdc466 clang::Parser::ParseTopLevelDecl(clang::OpaquePtr<clang::DeclGroupRef>&, clang::Sema::ModuleImportState&) (bin/clang-doc+0x17d1466)
#34 0x000056425ebd762e clang::ParseAST(clang::Sema&, bool, bool) (bin/clang-doc+0x17cc62e)
#35 0x000056425e91decf clang::FrontendAction::Execute() (bin/clang-doc+0x1512ecf)
#36 0x000056425e33e82d clang::CompilerInstance::ExecuteAction(clang::FrontendAction&) (bin/clang-doc+0xf3382d)
#37 0x000056425e2a906b clang::tooling::FrontendActionFactory::runInvocation(std::__2::shared_ptr<clang::CompilerInvocation>, clang::FileManager*, std::__2::shared_ptr<clang::PCHContainerOperations>, clang::DiagnosticConsumer*) (bin/clang-doc+0xe9e06b)
#38 0x000056425e2a8fc4 clang::tooling::ToolInvocation::runInvocation(char const*, clang::driver::Compilation*, std::__2::shared_ptr<clang::CompilerInvocation>, std::__2::shared_ptr<clang::PCHContainerOperations>) (bin/clang-doc+0xe9dfc4)
#39 0x000056425e2a7e48 clang::tooling::ToolInvocation::run() (bin/clang-doc+0xe9ce48)
#40 0x000056425e2aa99b clang::tooling::ClangTool::run(clang::tooling::ToolAction*) (bin/clang-doc+0xe9f99b)
#41 0x000056425e251ed5 void std::__2::__function::__policy_invoker<void ()>::__call_impl[abi:nn190000]<std::__2::__function::__default_alloc_func<std::__2::__bind<clang::tooling::AllTUsToolExecutor::execute(llvm::ArrayRef<std::__2::pair<std::__2::unique_ptr<clang::tooling::FrontendActionFactory, std::__2::default_delete<clang::tooling::FrontendActionFactory>>, std::__2::function<std::__2::vector<std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char>>, std::__2::allocator<std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char>>>> (std::__2::vector<std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char>>, std::__2::allocator<std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char>>>> const&, llvm::StringRef)>>>)::$_0, std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char>>&>, void ()>>(std::__2::__function::__policy_storage const*) AllTUsExecution.cpp:0:0
#42 0x000056425e1cddfe std::__2::__deferred_assoc_state<void, std::__2::__async_func<std::__2::function<void ()>>>::__execute() ClangDocMain.cpp:0:0
#43 0x000056425e20afec llvm::StdThreadPool::processTasks(llvm::ThreadPoolTaskGroup*) (bin/clang-doc+0xdfffec)
#44 0x000056425e20c65c void* llvm::thread::ThreadProxy<std::__2::tuple<llvm::StdThreadPool::grow(int)::$_0>>(void*) ThreadPool.cpp:0:0
#45 0x00007fa2db01c45c start_thread ./nptl/pthread_create.c:444:8
#46 0x00007fa2db09cbbc clone3 ./misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:83:0
[1]    290300 IOT instruction  bin/clang-doc --output=docs --executor=all-TUs ./compile_commands.json

cmake invocation:

cmake -GNinja -DCMAKE_EXPORT_COMPILE_COMMANDS=ON -DLLVM_ENABLE_PROJECTS=clang-tools-extra -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=ON ../llvm -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++

command run in the build directory:
bin/clang-doc --output=docs --executor=all-TUs ./compile_commands.json

I'll try and see if I can get it to repoduce w/ -gen-reproducer, but that may take a while.

[llvmbot]

@llvm/issue-subscribers-bug

Author: Paul Kirth (ilovepi)

When using clang-doc to generate documentation for LLVM, I'm seeing an assertion failure in Sema. ``` [1476/5643] Processing file /usr/local/google/home/paulkirth/llvm-project/llvm/lib/Support/FileCollector.cpp clang-doc: /usr/local/google/home/paulkirth/llvm-project/clang/lib/Sema/SemaExprMember.cpp:993: ExprResult clang::Sema::BuildMemberReferenceExpr(Expr *, QualType, SourceLocation, bool, const CXXScopeSpec &, SourceLocation, NamedDecl *, LookupResult &, const TemplateArgumentListInfo *, const Scope *, bool, ActOnMemberAccessExtraArgs *): Assertion `!SS.isInvalid() && "nested-name-specifier cannot be invalid"' failed. 1 error generated. #0 0x000056425e24eb88 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (bin/clang-doc+0xe43b88) #1 0x000056425e24c85c llvm::sys::RunSignalHandlers() (bin/clang-doc+0xe4185c) #2 0x000056425e24f240 SignalHandler(int) Signals.cpp:0:0 #3 0x00007fa2dafd0510 (/lib/x86_64-linux-gnu/libc.so.6+0x3c510) #4 0x00007fa2db01e16c __pthread_kill_implementation ./nptl/pthread_kill.c:44:76 #5 0x00007fa2dafd0472 raise ./signal/../sysdeps/posix/raise.c:27:6 #6 0x00007fa2dafba4b2 abort ./stdlib/abort.c:81:7 #7 0x00007fa2dafba3d5 _nl_load_domain ./intl/loadmsgcat.c:1177:9 #8 0x00007fa2dafc93a2 (/lib/x86_64-linux-gnu/libc.so.6+0x353a2) #9 0x000056425f538966 clang::Sema::BuildMemberReferenceExpr(clang::Expr*, clang::QualType, clang::SourceLocation, bool, clang::CXXScopeSpec const&, clang::SourceLocation, clang::NamedDecl*, clang::LookupResult&, clang::TemplateArgumentListInfo const*, clang::Scope const*, bool, clang::Sema::ActOnMemberAccessExtraArgs*) (bin/clang-doc+0x212d966) #10 0x000056425f535aa1 clang::Sema::BuildMemberReferenceExpr(clang::Expr*, clang::QualType, clang::SourceLocation, bool, clang::CXXScopeSpec&, clang::SourceLocation, clang::NamedDecl*, clang::DeclarationNameInfo const&, clang::TemplateArgumentListInfo const*, clang::Scope const*, clang::Sema::ActOnMemberAccessExtraArgs*) (bin/clang-doc+0x212aaa1) #11 0x000056425f539843 clang::Sema::ActOnMemberAccessExpr(clang::Scope*, clang::Expr*, clang::SourceLocation, clang::tok::TokenKind, clang::CXXScopeSpec&, clang::SourceLocation, clang::UnqualifiedId&, clang::Decl*) (bin/clang-doc+0x212e843) #12 0x000056425ec061da clang::Parser::ParsePostfixExpressionSuffix(clang::ActionResult<clang::Expr*, true>) (bin/clang-doc+0x17fb1da) #13 0x000056425ec08409 clang::Parser::ParseCastExpression(clang::Parser::CastParseKind, bool, bool&, clang::Parser::TypeCastState, bool, bool*) (bin/clang-doc+0x17fd409) #14 0x000056425ec02dec clang::Parser::ParseAssignmentExpression(clang::Parser::TypeCastState) (bin/clang-doc+0x17f7dec) #15 0x000056425ec02d19 clang::Parser::ParseExpression(clang::Parser::TypeCastState) (bin/clang-doc+0x17f7d19) #16 0x000056425eccc911 clang::Parser::ParseReturnStatement() (bin/clang-doc+0x18c1911) #17 0x000056425ecc5251 clang::Parser::ParseStatementOrDeclarationAfterAttributes(llvm::SmallVector<clang::Stmt*, 32u>&, clang::Parser::ParsedStmtContext, clang::SourceLocation*, clang::ParsedAttributes&, clang::ParsedAttributes&) (bin/clang-doc+0x18ba251) #18 0x000056425ecc36d2 clang::Parser::ParseStatementOrDeclaration(llvm::SmallVector<clang::Stmt*, 32u>&, clang::Parser::ParsedStmtContext, clang::SourceLocation*) (bin/clang-doc+0x18b86d2) #19 0x000056425eccf451 clang::Parser::ParseCompoundStatementBody(bool) (bin/clang-doc+0x18c4451) #20 0x000056425ecd0732 clang::Parser::ParseFunctionStatementBody(clang::Decl*, clang::Parser::ParseScope&) (bin/clang-doc+0x18c5732) #21 0x000056425ecfa080 clang::Parser::ParseLexedMethodDef(clang::Parser::LexedMethod&) (bin/clang-doc+0x18ef080) #22 0x000056425ecf863a clang::Parser::ParseLexedMethodDefs(clang::Parser::ParsingClass&) (bin/clang-doc+0x18ed63a) #23 0x000056425ec3a7a7 clang::Parser::ParseCXXMemberSpecification(clang::SourceLocation, clang::SourceLocation, clang::ParsedAttributes&, unsigned int, clang::Decl*) (bin/clang-doc+0x182f7a7) #24 0x000056425ec378e7 clang::Parser::ParseClassSpecifier(clang::tok::TokenKind, clang::SourceLocation, clang::DeclSpec&, clang::Parser::ParsedTemplateInfo const&, clang::AccessSpecifier, bool, clang::Parser::DeclSpecContext, clang::ParsedAttributes&) (bin/clang-doc+0x182c8e7) #25 0x000056425ec639cb clang::Parser::ParseDeclarationSpecifiers(clang::DeclSpec&, clang::Parser::ParsedTemplateInfo const&, clang::AccessSpecifier, clang::Parser::DeclSpecContext, clang::Parser::LateParsedAttrList*, clang::ImplicitTypenameContext) (bin/clang-doc+0x18589cb) #26 0x000056425ebdfb1c clang::Parser::ParseDeclOrFunctionDefInternal(clang::ParsedAttributes&, clang::ParsedAttributes&, clang::ParsingDeclSpec&, clang::AccessSpecifier) (bin/clang-doc+0x17d4b1c) #27 0x000056425ebdf78b clang::Parser::ParseDeclarationOrFunctionDefinition(clang::ParsedAttributes&, clang::ParsedAttributes&, clang::ParsingDeclSpec*, clang::AccessSpecifier) (bin/clang-doc+0x17d478b) #28 0x000056425ebde5a9 clang::Parser::ParseExternalDeclaration(clang::ParsedAttributes&, clang::ParsedAttributes&, clang::ParsingDeclSpec*) (bin/clang-doc+0x17d35a9) #29 0x000056425ec2cbff clang::Parser::ParseInnerNamespace(llvm::SmallVector<clang::Parser::InnerNamespaceInfo, 4u> const&, unsigned int, clang::SourceLocation&, clang::ParsedAttributes&, clang::BalancedDelimiterTracker&) (bin/clang-doc+0x1821bff) #30 0x000056425ec2beb7 clang::Parser::ParseNamespace(clang::DeclaratorContext, clang::SourceLocation&, clang::SourceLocation) (bin/clang-doc+0x1820eb7) #31 0x000056425ec5c9af clang::Parser::ParseDeclaration(clang::DeclaratorContext, clang::SourceLocation&, clang::ParsedAttributes&, clang::ParsedAttributes&, clang::SourceLocation*) (bin/clang-doc+0x18519af) #32 0x000056425ebde0fd clang::Parser::ParseExternalDeclaration(clang::ParsedAttributes&, clang::ParsedAttributes&, clang::ParsingDeclSpec*) (bin/clang-doc+0x17d30fd) #33 0x000056425ebdc466 clang::Parser::ParseTopLevelDecl(clang::OpaquePtr<clang::DeclGroupRef>&, clang::Sema::ModuleImportState&) (bin/clang-doc+0x17d1466) #34 0x000056425ebd762e clang::ParseAST(clang::Sema&, bool, bool) (bin/clang-doc+0x17cc62e) #35 0x000056425e91decf clang::FrontendAction::Execute() (bin/clang-doc+0x1512ecf) #36 0x000056425e33e82d clang::CompilerInstance::ExecuteAction(clang::FrontendAction&) (bin/clang-doc+0xf3382d) #37 0x000056425e2a906b clang::tooling::FrontendActionFactory::runInvocation(std::__2::shared_ptr<clang::CompilerInvocation>, clang::FileManager*, std::__2::shared_ptr<clang::PCHContainerOperations>, clang::DiagnosticConsumer*) (bin/clang-doc+0xe9e06b) #38 0x000056425e2a8fc4 clang::tooling::ToolInvocation::runInvocation(char const*, clang::driver::Compilation*, std::__2::shared_ptr<clang::CompilerInvocation>, std::__2::shared_ptr<clang::PCHContainerOperations>) (bin/clang-doc+0xe9dfc4) #39 0x000056425e2a7e48 clang::tooling::ToolInvocation::run() (bin/clang-doc+0xe9ce48) #40 0x000056425e2aa99b clang::tooling::ClangTool::run(clang::tooling::ToolAction*) (bin/clang-doc+0xe9f99b) #41 0x000056425e251ed5 void std::__2::__function::__policy_invoker<void ()>::__call_impl[abi:nn190000]<std::__2::__function::__default_alloc_func<std::__2::__bind<clang::tooling::AllTUsToolExecutor::execute(llvm::ArrayRef<std::__2::pair<std::__2::unique_ptr<clang::tooling::FrontendActionFactory, std::__2::default_delete<clang::tooling::FrontendActionFactory>>, std::__2::function<std::__2::vector<std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char>>, std::__2::allocator<std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char>>>> (std::__2::vector<std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char>>, std::__2::allocator<std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char>>>> const&, llvm::StringRef)>>>)::$_0, std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char>>&>, void ()>>(std::__2::__function::__policy_storage const*) AllTUsExecution.cpp:0:0 #42 0x000056425e1cddfe std::__2::__deferred_assoc_state<void, std::__2::__async_func<std::__2::function<void ()>>>::__execute() ClangDocMain.cpp:0:0 #43 0x000056425e20afec llvm::StdThreadPool::processTasks(llvm::ThreadPoolTaskGroup*) (bin/clang-doc+0xdfffec) #44 0x000056425e20c65c void* llvm::thread::ThreadProxy<std::__2::tuple<llvm::StdThreadPool::grow(int)::$_0>>(void*) ThreadPool.cpp:0:0 #45 0x00007fa2db01c45c start_thread ./nptl/pthread_create.c:444:8 #46 0x00007fa2db09cbbc clone3 ./misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:83:0 [1] 290300 IOT instruction bin/clang-doc --output=docs --executor=all-TUs ./compile_commands.json ```

cmake invocation:

cmake -GNinja -DCMAKE_EXPORT_COMPILE_COMMANDS=ON -DLLVM_ENABLE_PROJECTS=clang-tools-extra -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=ON ../llvm -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++

command run in the build directory:
bin/clang-doc --output=docs --executor=all-TUs ./compile_commands.json

I'll try and see if I can get it to repoduce w/ -gen-reproducer, but that may take a while.

[ilovepi]

cc: @PeterChou1

[ilovepi]

A few things I've run into investigating this:

1. In my latest run it didn't hit the assertion. This implies its something racy or maybe address dependent (maybe an out of bounds read/write? TSAN or ASAN are likely to help diagnose this particular issue.
2. it took 3 hours 20 minutes to run, which is excessive, but even then it OOM'd on a machine w/ 177GB of ram and ~ 2GB swap.

IMO this should never happen, so something is very wrong here, and we're either leaking memory, have some bad allocation pattern, or cycle somewhere in either a data structure or in the code. The OOM happens in the Reduce step, which is basically merging the Documentation data structures from across TUs. It's possible that the bug causig the OOM exists there and we've used an inappropriate data structure or algorithm that fails to scale appropriately. Do note though, that the OOM and the assertion don't seem to be related.

[PeterChou1]

I did some more digging and I don't think the assertion has to do with clang-doc here's a minimal libtooling file.

#include "clang/Frontend/FrontendActions.h"
#include "clang/Tooling/CommonOptionsParser.h"
#include "clang/Tooling/Tooling.h"
#include "llvm/Support/CommandLine.h"
#include "clang/Tooling/Execution.h"

using namespace clang::tooling;
using namespace llvm;

class MapASTVisitor : public clang::ASTConsumer {
public:
  explicit MapASTVisitor() {}
  void HandleTranslationUnit(clang::ASTContext &Context) override {}
};
class MyAction : public clang::ASTFrontendAction {
public:
  MyAction() {}
  std::unique_ptr<clang::ASTConsumer>
  CreateASTConsumer(clang::CompilerInstance &Compiler,
                    llvm::StringRef InFile) override {
    return std::make_unique<MapASTVisitor>();
  }
};

static llvm::cl::OptionCategory MyToolCategory("my-tool");
int main(int argc, const char **argv) {
  const char *Overview = "";
  auto Executor = clang::tooling::createExecutorFromCommandLineArgs(
      argc, argv, MyToolCategory, Overview);
  if (!Executor) {
    llvm::errs() << toString(Executor.takeError()) << "\n";
    return 1;
  }
  auto Err = Executor->get()->execute(newFrontendActionFactory<MyAction>());
  return 0;
}

running this against llvm compile_commands.json resulted in the same assertion error being generated

[ilovepi]

does the error reproduce when you don't use the exectuor? https://clang.llvm.org/docs/LibASTMatchersTutorial.html has a few examples.

I also wonder if the same issue reproduces with the existing actions, like syntaxonly when run w/ the executor? I have a suspicion that there could be a data race that only happens when run w/ the executor.

[PeterChou1]

This appears to a bug in lib tooling where it's is failing to find a file which breaks and assumption in Sema, which triggers an assertion.

[ilovepi]

This appears to a bug in lib tooling where it's is failing to find a file which breaks and assumption in Sema, which triggers an assertion.

When updating issues, please include sufficient context for others to follow.

The underlying problem was that a header path was set incorrectly in the compile_commands.json. That's a different kind of bug on its own, but we don't control how build systems like CMake or Ninja generate that file. Once the header is missing, the source is in an incomplete state, and the assertion trips (I assume, since a scope isn't terminated as expected based on the assert message).

So, this isn't a bug in libTooling. Its a bug in clang that the assertion is reachable after a file is missing, and the AST being analyzed is in some incorrect/invalid state. IMO the missing file should be fatal, and should prevent the assertion from firing. I'm not sure what mechanisms we have to address that, on the clang side, but this isn't a bug in clang-doc, nor in libTooling.

@AaronBallman do you know if we have some handling/short circuit mechanism to prevent assertions like that from firing. Also, do you know who we should loop in? I'm not really familiar enough w/ clang internals/code owners to know who's a good point of contact here besides you or Eli ...

[AaronBallman]

This appears to a bug in lib tooling where it's is failing to find a file which breaks and assumption in Sema, which triggers an assertion.

When updating issues, please include sufficient context for others to follow.

The underlying problem was that a header path was set incorrectly in the compile_commands.json. That's a different kind of bug on its own, but we don't control how build systems like CMake or Ninja generate that file. Once the header is missing, the source is in an incomplete state, and the assertion trips (I assume, since a scope isn't terminated as expected based on the assert message).

So, this isn't a bug in libTooling. Its a bug in clang that the assertion is reachable after a file is missing, and the AST being analyzed is in some incorrect/invalid state. IMO the missing file should be fatal, and should prevent the assertion from firing. I'm not sure what mechanisms we have to address that, on the clang side, but this isn't a bug in clang-doc, nor in libTooling.

The way fatal diagnostics work in Clang is that they suppress emitting additional diagnostics but don't (generally) cause us to stop processing things entirely. For example: https://godbolt.org/z/115hhq4vr you'll see that we emit a fatal error for the unknown header file, no diagnostics for the unknown identifiers, but we do still cause a crash from the debugging pragma.

That said, the diagnostics engine does have APIs to say whether a fatal or unrecoverable error has occurred so we can early return in some cases (e.g., cut off printing a bunch of notes after reaching -ferror-limit=, etc).

@AaronBallman do you know if we have some handling/short circuit mechanism to prevent assertions like that from firing. Also, do you know who we should loop in? I'm not really familiar enough w/ clang internals/code owners to know who's a good point of contact here besides you or Eli ...

CC @sdkrystian who authored 8009bbe which added this assertion.

[ilovepi]

The way fatal diagnostics work in Clang is that they suppress emitting additional diagnostics but don't (generally) cause us to stop processing things entirely. For example: https://godbolt.org/z/115hhq4vr you'll see that we emit a fatal error for the unknown header file, no diagnostics for the unknown identifiers, but we do still cause a crash from the debugging pragma.

That said, the diagnostics engine does have APIs to say whether a fatal or unrecoverable error has occurred so we can early return in some cases (e.g., cut off printing a bunch of notes after reaching -ferror-limit=, etc).

Ah, thanks for the context. Looking at the stack trace, I'm not really sure where an early return would need to be. Likely there's quite a few places that would make sense. I'll try to set aside some time later in the week to look into that.

[sdkrystian]

@ilovepi @AaronBallman This will be fixed by #98167.