[Qt-creator] Issue parsing QStringLiteral() in SpellChecker Plugin

[Carel Combrink]

Hi Guys,

I saw that my SpellChecker Plugin does not handle string literals enclosed
in QStringLiteral(). correctly. I am pretty sure it is because
QStringLiteral() is a macro that expands to the actual code to handle the
literal.

The issue with this is, that I need to parse the string to check for
spelling mistakes and then if there are mistakes, underline the misspelled
word and give the user the options to replace the misspelled word.

Due to the expansion of the macro, I am finding it rather hard to get the
position of the words as the user sees it, not my plugin. For example if I
have a mistake on "line: 8, Col: 63" (reported by QtC), my parser sees 2
instances of the word at (10:65) and (10:232).

How do I extract the 'correct location' (Location in the IDE) for the
QStringLiteral() expansion?

I have started searching and looked a bit at the WrapStringLiteral class
(part of CppQuickfixes) and it seems like the WrapStringLiteral::analyze()
function can do what I need. I am not sure how to use this, or implement
something similar.

Any help will be greatly appreciated.

PS: I will also be very grateful if this solution can be used to also know
which strings are translatable strings (enclosed in tr()). I would like to
add the option to my plugin to let the user request that only string
literals that will be translated be parsed.

---
Check out my SpellChecker Plugin for Qt Creator on github @
https://github.com/CJCombrink/SpellChecker-Plugin
Regards,
Carel

[Typz]

Hello,

At the moment, there are two "levels" of preprocessing: full preprocessor is used for code analysis, while "fast preprocessor" is used for highlighting...

With fast preprocessor, QStringLiteral should not be expanded. So just running fast preprocessor on the source code before parsing strings should solve the problem.

As for matching position, I remember there used to be some functions in c++ editor, which are used for highlighting macros... That was some time ago though, it may have changed since...

Congratulations for making this great plugin, Best Regards,
--
Francois

Post by Carel Combrink
Hi Guys,
I saw that my SpellChecker Plugin does not handle string literals enclosed in QStringLiteral(). correctly. I am pretty sure it is because QStringLiteral() is a macro that expands to the actual code to handle the literal.
The issue with this is, that I need to parse the string to check for spelling mistakes and then if there are mistakes, underline the misspelled word and give the user the options to replace the misspelled word.
Due to the expansion of the macro, I am finding it rather hard to get the position of the words as the user sees it, not my plugin. For example if I have a mistake on "line: 8, Col: 63" (reported by QtC), my parser sees 2 instances of the word at (10:65) and (10:232).
How do I extract the 'correct location' (Location in the IDE) for the QStringLiteral() expansion?
I have started searching and looked a bit at the WrapStringLiteral class (part of CppQuickfixes) and it seems like the WrapStringLiteral::analyze() function can do what I need. I am not sure how to use this, or implement something similar.
Any help will be greatly appreciated.
PS: I will also be very grateful if this solution can be used to also know which strings are translatable strings (enclosed in tr()). I would like to add the option to my plugin to let the user request that only string literals that will be translated be parsed.
---
Regards,
Carel
_______________________________________________
Qt-creator mailing list
http://lists.qt-project.org/mailman/listinfo/qt-creator

[Nikolai Kosjar]

Hiya!

Post by Carel Combrink
Due to the expansion of the macro, I am finding it rather hard to get the position of the words as the user sees it, not my plugin. For example if I have a mistake on "line: 8, Col: 63" (reported by QtC), my parser sees 2 instances of the word at (10:65) and (10:232).

I don't know how you get the line and column information, but you should use
the get* functions of CPlusPlus::TranslationUnit for this.

Consider to what QStringLiteral("hello there") might expand to:

auto s = ([]() -> QString { enum { Size = sizeof(u"" "hello there")/2 - 1 }; static const QStaticStringData<Size> qstring_literal = { { { { (-1) } }, Size, 0, 0, sizeof(QStringData) }, u"" "hello there" }; QStringDataPtr holder = { qstring_literal.data_ptr() }; const QString qstring_literal_temp(holder); return qstring_literal_temp; }());

The following code will report 4 string literals (note the u""), assuming that
document comes from CppTools::CppModelManager::documentUpdated():

if (TranslationUnit *translationUnit = document->translationUnit()) {
for (unsigned i = 0; i < translationUnit->tokenCount(); ++i) {
const Token t = translationUnit->tokenAt(i);
if (t.isStringLiteral() && t.expanded()) {
unsigned line, column;
translationUnit->getTokenStartPosition(i, &line, &column);
qDebug() << "String:" << t.spell() << "at line:" << line << "column:" << column;
}
}
}

output:
23 String: at line: 9 column: 41
24 String: hello there at line: 7 column: 29
63 String: at line: 9 column: 177
64 String: hello there at line: 7 column: 29

That's OK. You just have to ignore the duplicates. And note that u"" is a token
consisting of 3 utf16chars and you probably also want to ignore this one.

Post by Carel Combrink
I have started searching and looked a bit at the WrapStringLiteral class (part of CppQuickfixes) and it seems like the WrapStringLiteral::analyze() function can do what I need. I am not sure how to use this, or implement something similar.

Note that the quickfixes do operate on a slightly different document than the
one that is emitted via CppModelManager::documentUpdated(). So you can't use
that code.

Post by Carel Combrink
PS: I will also be very grateful if this solution can be used to also know which strings are translatable strings (enclosed in tr()).

The fast/simple solution that will work most of the time in Qt projects: Check
for the "tr" and "(" tokens before the string literal token. Note that it will
already fail for e.g.: MyClass tr("lala");

The slower/correct solution involves:
1. Finding the CallAST for tr(). Choose one of:
1.a Use an ASTVisitor for this and override visit(CallAST *ast)
See CheckSymbols::visit(CallAST *ast) for an example.
1.b Use ASTPath - see how it's used in cppquickfixes.cpp
See how ASTPath is used in cppquickfixes.cpp ("interface.path()").
2. Making a lookup on the "tr" to ensure it's the right one.

Remove the optimization line in src/libs/cplusplus/cplusplus.pro and rebuild
for a better debugging experience.

Nikolai

[Carel Combrink]

First of all, thank you for taking the time to respond to my issue.

Regarding the code you have supplied, it is almost the same code that I am
using, but I am not getting the same result as you. Is there perhaps a
reason why this would be?

I don't know how you get the line and column information, but you should use

Post by Nikolai Kosjar
the get* functions of CPlusPlus::TranslationUnit for this.

If you look at the original code on github for cppdocumentparser.cpp
<https://github.com/CJCombrink/SpellChecker-Plugin/blob/master/Parsers/CppParser/cppdocumentparser.cpp>
you
can check if I am doing the correct thing:

The following code will report 4 string literals (note the u""), assuming

Post by Nikolai Kosjar
that

line 83
<https://github.com/CJCombrink/SpellChecker-Plugin/blob/master/Parsers/CppParser/cppdocumentparser.cpp#L83>:
Connects to &CppTools::CppModelManager::documentUpdated
This then ends up calling CppDocumentParser::parseCppDocument
<https://github.com/CJCombrink/SpellChecker-Plugin/blob/master/Parsers/CppParser/cppdocumentparser.cpp#L186>
on line 186 if you follow the code for a valid file.

Post by Nikolai Kosjar
if (TranslationUnit *translationUnit = document->translationUnit()) {
for (unsigned i = 0; i < translationUnit->tokenCount(); ++i) {
const Token t = translationUnit->tokenAt(i);
if (t.isStringLiteral() && t.expanded()) {
unsigned line, column;
translationUnit->getTokenStartPosition(i, &line, &column);
qDebug() << "String:" << t.spell() << "at line:" << line <<
"column:" << column;
}
}
}

On line 211
<https://github.com/CJCombrink/SpellChecker-Plugin/blob/master/Parsers/CppParser/cppdocumentparser.cpp#L211>
you get to the relevant code in my parser. I previously had the issue thus
for that version of the code I just skip the expanded tokens.

So I have now updated the relevant code locally and the code can be seen in
the attached screenshots:
issue_source.png: Shows the source file that my SpellChecker is parsing.
Note that the cursor is @ Line 6, Column 32
issue_code.png: Shows my code along with my qDebug() outputs at the bottom.
Note that the debug reports the literal @ Line 8, Column 41.

It seems to me that my code always reports both instances to start at line
'real' + 2 and column at some 'random' constant column, no matter where I
move the string literal.

Note that the quickfixes do operate on a slightly different document than

Post by Nikolai Kosjar
the
one that is emitted via CppModelManager::documentUpdated(). So you can't use
that code.

Thanks for this information.

Post by Nikolai Kosjar
1.a Use an ASTVisitor for this and override visit(CallAST *ast)
See CheckSymbols::visit(CallAST *ast) for an example.
1.b Use ASTPath - see how it's used in cppquickfixes.cpp
See how ASTPath is used in cppquickfixes.cpp ("interface.path()").
2. Making a lookup on the "tr" to ensure it's the right one.

I have looked at this quickly but not in depth yet, for now the
StringLiteral is my priority so i will focus on trying to get that solved.
Perhaps then if I have issues I will ask about tr() strings. Thank you for
this information so long.

Just to mention my environment:
Windows 7 x64
Building with Qt 5.4 using msvc 2013
Qt Creator sources version: 3.4.82 (Does not mean to much I think)
The git commit version: ((463566a...)) on Jul 24 from
https://github.com/qtproject/qt-creator.git
I have also just updated to latest commit (50e9ef6...) but no significant
changes...

Regards,
Carel

[Nikolai Kosjar]

Hi!

Regarding the code you have supplied, it is almost the same code that I am using, but I am not getting the same result as you. Is there perhaps a reason why this would be?

Looks like a bug on Windows - there I also get the +2 lines difference.

Please create a minimal failing example and report at https://bugreports.qt.io/

Nikolai

[Carel Combrink]

Hi Kikolai,

Looks like a bug on Windows - there I also get the +2 lines difference.
I agree with you, on Linux I was able to implement your suggestion and now
my SpellChecker works correctly for QStringLiterals. PS: I have no way to
test on Mac so the code will be enabled if building from source.

Post by Nikolai Kosjar
Please create a minimal failing example and report at
https://bugreports.qt.io/

I have disabled this code in Windows but created a bug report @
https://bugreports.qt.io/browse/QTCREATORBUG-14834

Thank you very much for the help in this regard.
FYI: The updated code can be seen in the commit
<https://github.com/CJCombrink/SpellChecker-Plugin/commit/e4be1c71b6cb3ce2a34638e788477c415dae7f9d>
for
the fix.

I will probably only make a new release closer to the release of Qt Creator
3.5, perhaps after the RC.

Regards,