[Qt-creator] SyntaxHighlighting + Bison/FLex

[JeremY Larrieu]

Hello,

I'm working on making a plugin to support a specific language: Anubis.

I've started the highlighting part and I saw that QtCreator is making
syntax highlighting line by line instead of making it "globally" for the
whole file.

I just wanna know, how can I use the Bison/Flex files, used to check syntax
for Anubis language, to make syntax highlighting in QtCreator.
Knowing that most of the tokens declared in the Flex file can be multiline,
making a lexer, working line by line, able to detect those tokens is hard
and the code is too "verbose".

I've tried to adapt what I've found in Bison/Flex files to make a syntax
highlighter, but it's really painful and it makes further updates harder.
If I was able to apply highlighting on the whole file (not line by line),
it were simpler and I could use it to make some other functionalities at
the same time: code completion, code analysis, symbol detection, ...

Do you have an idea on how I can make syntax highlighting without rewriting
a full lexer ?

Thanks in advance.

Jeremy

[David Schulz]

Hi Jeremy,

That sounds more like a semantic highlighter to me. Have a look at
SemanticHighlighter::incrementalApplyExtraAdditionalFormats in
src\plugins\texteditor\semantichighlighter.cpp and the usages in the
QML/JS and C++ semantic highlighter.


Br,

David

Post by JeremY Larrieu
Hello,
I'm working on making a plugin to support a specific language: Anubis.
I've started the highlighting part and I saw that QtCreator is making
syntax highlighting line by line instead of making it "globally" for
the whole file.
I just wanna know, how can I use the Bison/Flex files, used to check
syntax for Anubis language, to make syntax highlighting in QtCreator.
Knowing that most of the tokens declared in the Flex file can be
multiline, making a lexer, working line by line, able to detect those
tokens is hard and the code is too "verbose".
I've tried to adapt what I've found in Bison/Flex files to make a
syntax highlighter, but it's really painful and it makes further
updates harder.
If I was able to apply highlighting on the whole file (not line by
line), it were simpler and I could use it to make some other
functionalities at the same time: code completion, code analysis,
symbol detection, ...
Do you have an idea on how I can make syntax highlighting without
rewriting a full lexer ?
Thanks in advance.
Jeremy
_______________________________________________
Qt-creator mailing list
http://lists.qt-project.org/mailman/listinfo/qt-creator

[Eike Ziller]

Post by JeremY Larrieu
Hello,
I'm working on making a plugin to support a specific language: Anubis.
I've started the highlighting part and I saw that QtCreator is making syntax highlighting line by line instead of making it "globally" for the whole file.
I just wanna know, how can I use the Bison/Flex files, used to check syntax for Anubis language, to make syntax highlighting in QtCreator.
Knowing that most of the tokens declared in the Flex file can be multiline, making a lexer, working line by line, able to detect those tokens is hard and the code is too "verbose".
I've tried to adapt what I've found in Bison/Flex files to make a syntax highlighter, but it's really painful and it makes further updates harder.
If I was able to apply highlighting on the whole file (not line by line), it were simpler and I could use it to make some other functionalities at the same time: code completion, code analysis, symbol detection, ...
Do you have an idea on how I can make syntax highlighting without rewriting a full lexer ?

Usually what you would do is to lex (or create an AST or whatever) the full document on document load and document change (probably asynchronously!), and make the highlighter just access that information based on the line it is supposed to highlight.

Post by JeremY Larrieu
Thanks in advance.
Jeremy
_______________________________________________
Qt-creator mailing list
http://lists.qt-project.org/mailman/listinfo/qt-creator

--
Eike Ziller
Principal Software Engineer

The Qt Company GmbH
Rudower Chaussee 13
D-12489 Berlin
***@qt.io
http://qt.io
Geschäftsführer: Mika Pälsi,
Juha Varelius, Mika Harjuaho
Sitz der Gesellschaft: Berlin, Registergericht: Amtsgericht Charlottenburg, HRB 144331 B

[Nikolai Kosjar]

Post by JeremY Larrieu
Hello,
I'm working on making a plugin to support a specific language: Anubis.
I've started the highlighting part and I saw that QtCreator is making
syntax highlighting line by line instead of making it "globally" for the
whole file.
I just wanna know, how can I use the Bison/Flex files, used to check
syntax for Anubis language, to make syntax highlighting in QtCreator.
Knowing that most of the tokens declared in the Flex file can be
multiline, making a lexer, working line by line, able to detect those
tokens is hard and the code is too "verbose".
I've tried to adapt what I've found in Bison/Flex files to make a syntax
highlighter, but it's really painful and it makes further updates harder.
If I was able to apply highlighting on the whole file (not line by
line), it were simpler and I could use it to make some other
functionalities at the same time: code completion, code analysis, symbol
detection, ...
Do you have an idea on how I can make syntax highlighting without
rewriting a full lexer ?
Thanks in advance.
Jeremy

Hi!

I guess you've found TextEditor::SyntaxHighlighter, which does the line
by line highlighting with highlightBlock(). For C++ and multi-line
tokens (e.g. C comments) we call our custom lexer for the current line
with the lexer state from the line before. If flex provides such a state
based yylex() equivalent, this might work.

The alternative is to use
SemanticHighlighter::incrementalApplyExtraAdditionalFormats(). With this
one, you can for example start parsing in a worker thread and provide
the highlighting information as a stream of
TextEditor::HighlightingResult items.

Nikolai

[JeremY Larrieu]

Hello,

Thanks for your answers.

I will look to semantic highlighting.
The purpose seems to fit my needs: syntax highlighting, symbol detection,
... each time a file is opened or modified.

@Nikolai the lex file works with at most 4/5 states and was not made to
work line by line. Modifying it in this direction will not help me to stick
closer to the language when new features will come. I will have to make too
many updates to my lex file each time the one's coming with the language is
updated.

Thanks again.

Jeremy

Post by JeremY Larrieu
Hello,
I'm working on making a plugin to support a specific language: Anubis.
I've started the highlighting part and I saw that QtCreator is making
syntax highlighting line by line instead of making it "globally" for the
whole file.
I just wanna know, how can I use the Bison/Flex files, used to check
syntax for Anubis language, to make syntax highlighting in QtCreator.
Knowing that most of the tokens declared in the Flex file can be
multiline, making a lexer, working line by line, able to detect those
tokens is hard and the code is too "verbose".
I've tried to adapt what I've found in Bison/Flex files to make a syntax
highlighter, but it's really painful and it makes further updates harder.
If I was able to apply highlighting on the whole file (not line by line),
it were simpler and I could use it to make some other functionalities at
the same time: code completion, code analysis, symbol detection, ...
Do you have an idea on how I can make syntax highlighting without
rewriting a full lexer ?
Thanks in advance.
Jeremy

Hi!
I guess you've found TextEditor::SyntaxHighlighter, which does the line by
line highlighting with highlightBlock(). For C++ and multi-line tokens
(e.g. C comments) we call our custom lexer for the current line with the
lexer state from the line before. If flex provides such a state based
yylex() equivalent, this might work.
The alternative is to use SemanticHighlighter::increment
alApplyExtraAdditionalFormats(). With this one, you can for example start
parsing in a worker thread and provide the highlighting information as a
stream of TextEditor::HighlightingResult items.
Nikolai

[JeremY Larrieu]

Hello

@Eike that's exactly what I wanna do. But I don't know where I can connect
tout this signals and the "highlightBlock" method doesn't provide the line
number to highlight nor from which file the line comes from.

Jeremy

Post by JeremY Larrieu
Hello,
Thanks for your answers.
I will look to semantic highlighting.
The purpose seems to fit my needs: syntax highlighting, symbol detection,
... each time a file is opened or modified.
@Nikolai the lex file works with at most 4/5 states and was not made to
work line by line. Modifying it in this direction will not help me to stick
closer to the language when new features will come. I will have to make too
many updates to my lex file each time the one's coming with the language is
updated.
Thanks again.
Jeremy

Post by Nikolai Kosjar

Post by JeremY Larrieu
Hello,
I'm working on making a plugin to support a specific language: Anubis.
I've started the highlighting part and I saw that QtCreator is making
syntax highlighting line by line instead of making it "globally" for the
whole file.
I just wanna know, how can I use the Bison/Flex files, used to check
syntax for Anubis language, to make syntax highlighting in QtCreator.
Knowing that most of the tokens declared in the Flex file can be
multiline, making a lexer, working line by line, able to detect those
tokens is hard and the code is too "verbose".
I've tried to adapt what I've found in Bison/Flex files to make a syntax
highlighter, but it's really painful and it makes further updates harder.
If I was able to apply highlighting on the whole file (not line by
line), it were simpler and I could use it to make some other
functionalities at the same time: code completion, code analysis, symbol
detection, ...
Do you have an idea on how I can make syntax highlighting without
rewriting a full lexer ?
Thanks in advance.
Jeremy

Hi!
I guess you've found TextEditor::SyntaxHighlighter, which does the line
by line highlighting with highlightBlock(). For C++ and multi-line tokens
(e.g. C comments) we call our custom lexer for the current line with the
lexer state from the line before. If flex provides such a state based
yylex() equivalent, this might work.
The alternative is to use SemanticHighlighter::increment
alApplyExtraAdditionalFormats(). With this one, you can for example
start parsing in a worker thread and provide the highlighting information
as a stream of TextEditor::HighlightingResult items.
Nikolai

[Nikolai Kosjar]

Hi!

Within your derived TextEditor::SyntaxHighlighter class, you should be
able to access currentBlock().blockNumber() - if you document does not
have any fancy layout items (images, tables), that one should correspond
to the line number (maybe off by one, don't remember).

You probably would also derive from TextEditor::TextDocument. From
there, you can start your async lexing/parsing operation if the document
is opened or changed. Connect to IDocument::filePathChanged and
Core::IDocument::contentsChanged. See e.g. CppEditorDocument for details
(this is more than you need, your version would look much simpler).

Nikolai

Post by JeremY Larrieu
Hello
@Eike that's exactly what I wanna do. But I don't know where I can
connect tout this signals and the "highlightBlock" method doesn't
provide the line number to highlight nor from which file the line comes
from.
Jeremy
Hello,
Thanks for your answers.
I will look to semantic highlighting.
The purpose seems to fit my needs: syntax highlighting, symbol
detection, ... each time a file is opened or modified.
@Nikolai the lex file works with at most 4/5 states and was not made
to work line by line. Modifying it in this direction will not help
me to stick closer to the language when new features will come. I
will have to make too many updates to my lex file each time the
one's coming with the language is updated.
Thanks again.
Jeremy
Hello,
I'm working on making a plugin to support a specific
language: Anubis.
I've started the highlighting part and I saw that QtCreator
is making syntax highlighting line by line instead of making
it "globally" for the whole file.
I just wanna know, how can I use the Bison/Flex files, used
to check syntax for Anubis language, to make syntax
highlighting in QtCreator.
Knowing that most of the tokens declared in the Flex file
can be multiline, making a lexer, working line by line, able
to detect those tokens is hard and the code is too "verbose".
I've tried to adapt what I've found in Bison/Flex files to
make a syntax highlighter, but it's really painful and it
makes further updates harder.
If I was able to apply highlighting on the whole file (not
line by line), it were simpler and I could use it to make
some other functionalities at the same time: code
completion, code analysis, symbol detection, ...
Do you have an idea on how I can make syntax highlighting
without rewriting a full lexer ?
Thanks in advance.
Jeremy
Hi!
I guess you've found TextEditor::SyntaxHighlighter, which does
the line by line highlighting with highlightBlock(). For C++ and
multi-line tokens (e.g. C comments) we call our custom lexer for
the current line with the lexer state from the line before. If
flex provides such a state based yylex() equivalent, this might
work.
The alternative is to use
SemanticHighlighter::incrementalApplyExtraAdditionalFormats().
With this one, you can for example start parsing in a worker
thread and provide the highlighting information as a stream of
TextEditor::HighlightingResult items.
Nikolai