Preprocessing and scintilla_22

[eranif]

The work to do is RefactoringEngine::GetInterestingMacros(text), which is empty right now.

Yea, I forgot i had a place holder for this

Eran

[jfouche]

You should not need to change anything, it should all be set already.

Open the workspace under PreProcessor/

select the PP.workspace open it hit F7 and its done (including cpoying all relevant files)

The yacc commands are already there (they are under 'custom makefile steps') + the post commands

Yes, you're right What did I miss when I saw those build error first time I tried ? Sorry. But, at least, I learned more about lex and yacc.

[jfouche]

I suggest we change the signature of the GetInterestingMacros to :

Code: Select all

void LEditor::GetInterestingMacros();

in order to do it asynchronously.
The design, as you told me, should be as following :

GetInterestingMacros :
- Send a ParseRequest (PR_PARSEINCLUDES) of the current file, in order to have all headers.

OnIncludeFilesScanDone :
- PPScan (not for CC) of all included files.
- PPScan (not for CC) of the current file.
- Apply PPTable::Instance()->Export() to keyword 4 of scintilla

In order to achieve this, we need also to create a thread request to do the step 2.

What do you think ?

[eranif]

in order to do it asynchronously.
The design, as you told me, should be as following :

GetInterestingMacros :
- Send a ParseRequest (PR_PARSEINCLUDES) of the current file, in order to have all headers.

OnIncludeFilesScanDone :
- PPScan (not for CC) of all included files.
- PPScan (not for CC) of the current file.
- Apply PPTable::Instance()->Export() to keyword 4 of scintilla

No. Just add new request type to the parse and let it do all the rest.
PPScan should be only called from the parser thread (one excpetion is the 'Tags Settings dialog | Parse! button)

Add new event to the parser thread and post it + the result back to the main app.

Important note:
wxString is NOT thread safe, so use wxChar* incase you need to pass string content from the parser thread to the app and all the content should be allocated on the heap

Eran

[jfouche]

I think we will encouter side effects due to the fact that PPScan is very simple, and doesn't care if we are in a valid block or not. We let scintilla do the hard work. I was thinking about a wxWidgets based application. I suppose all wx macros will be defined, so if I write someting like :

Code: Select all

#ifdef __WXMSW__
  ...
#else
  ...
#endif

the first block will be always normal (even if I'm under linux), and the second will be always grayed.

A solution may be to add some per configuration options to allow to ignore some definitions, and maybe an other to add some more.

[jfouche]

Well, bad news...
I have a very big problem of performances : the macro string is about 550 kB for a VersionManager plugin file (small workspace but I have MinGW and wxWidgets includes available). I think scintilla doesn't like it.
Do you have an idea on how to impove this ? Maybe only allow project file parsing (but it won't be as good as if we take all include files). Or a better macro finder grammar in case we don't want it for CC (As I understood the pp lex and yacc files, I can probably improve it for this use).
I go to sleep now, without success

[eranif]

Ok, now I remembered what I wanted to do :

• Parse the current file without any of its include fiels for the interesting macros (macros which are refered by #if/#ifdef/#ifndef (make sure that we don't pass the blockguard macro)) - this was the purpose of 'GetInterstingMacros' - lets call this list of macros 'group 1'
• Call PPSCan in the parse_thread to collect all macros (from include files as well) - lets call this macros 'group 2'
• Pass to scintilla all macros which exists in both groups

You should have no performance problem taking this approah
Eran

[jfouche]

good idea
Gonna test it this afternoon

[jfouche]

I change a little bit the pp parser in order to manage #if / #ifdef / #ifndef macro. Moreover, I add a method in PPTable to allow to add used macros (found by the change below). I'm interrested by a code review, to know if I'm taking the good way to solve our problem :

Code: Select all

Index: pp.l
===================================================================
--- pp.l	(revision 4529)
+++ pp.l	(working copy)
@@ -16,6 +16,7 @@
 %x define_generic_c_comment
 %x define_generic_cpp_comment
 %x ifdef_state
+%x if_state
 
 %option yylineno
 
@@ -84,14 +85,16 @@
 
 "L"?[']{c_char}+[']     {/* eat a string */}
 "L"?["]{s_char}*["]     {/* eat a string */}
+
 #                           { BEGIN(PP);                                          }
+
 <PP>define                  { BEGIN(define_state); RET_VAL(PP_DEFINE);            }
-<PP>if                      { if(in_if_1) in_if_1++; RET_VAL(PP_IF);              }
+<PP>if                      { if(in_if_1) in_if_1++; BEGIN(if_state); RET_VAL(PP_IF);              }
 <PP>0                       { RET_VAL(PP_ZERO);                                   }
 <PP>__cplusplus             { RET_VAL(PP_CPLUSPLUS);                              }
-<PP>ifdef                   { if(in_if_1) in_if_1++;BEGIN(ifdef_state); RET_VAL(PP_IFDEF);              }
+<PP>ifdef                   { if(in_if_1) in_if_1++; BEGIN(ifdef_state); RET_VAL(PP_IFDEF);              }
 <PP>defined                 { RET_VAL(PP_DEFINED);                                }
-<PP>ifndef                  { if(in_if_1) in_if_1++;RET_VAL(PP_IFNDEF);           }
+<PP>ifndef                  { if(in_if_1) in_if_1++; BEGIN(ifdef_state); RET_VAL(PP_IFNDEF);           }
 <PP>undef                   { RET_VAL(PP_UNDEF);                                  }
 <PP>else                    { if(in_if_1 == 1) in_if_1 = 0;RET_VAL(PP_ELSE);      }
 <PP>elif                    { if(in_if_1 == 1) in_if_1--;RET_VAL(PP_ELIF);        }
@@ -99,16 +102,26 @@
 <PP>include                 { BEGIN(incl); RET_VAL(PP_INCLUDE);                   }
 <PP>\n                      { BEGIN(INITIAL);                                     }
 <PP>.                       {}
+
 <ifdef_state>__cplusplus    { RET_VAL(PP_CPLUSPLUS);                              }
 <ifdef_state>{identifier}   { RET_VAL(PP_IDENTIFIER);                             }
 <ifdef_state>"("            { RET_VAL(((int)'('));                                }
 <ifdef_state>")"            { RET_VAL(((int)')'));                                }
 <ifdef_state>\n             { BEGIN(INITIAL);                                     }
 <ifdef_state>.              {}
+
+<if_state>__cplusplus    { RET_VAL(PP_CPLUSPLUS);                              }
+<if_state>{identifier}   { RET_VAL(PP_IDENTIFIER);                             }
+<if_state>"("            { RET_VAL(((int)'('));                                }
+<if_state>")"            { RET_VAL(((int)')'));                                }
+<if_state>\n             { BEGIN(INITIAL);                                     }
+<if_state>.              {}
+
 <define_state>{identifier}  { BEGIN(define_state_2); RET_VAL(PP_IDENTIFIER); }
 <define_state>\\[\n\r]{1,2} { /* continue define_state */}
 <define_state>\n            { BEGIN(INITIAL);}
 <define_state>.             {}
+
 <define_state_2>[ \t]       {
 /* whitespaces are NOT allowed between the signature and the macro's name, 
  * if we found a whitespace, handle this macro as a simple macro */
@@ -125,10 +138,12 @@
 <define_state_2>"("         { BEGIN(define_state_signature); RET_VAL(((int)'(')); }
 <define_state_2>\n          { BEGIN(INITIAL); _definition.clear(); g_definition.Clear(); RET_VAL(PP_COMPLEX_REPLACEMENT);}
 <define_state_2>.           { BEGIN(define_state_definition);   _definition = yytext;}
+
 <define_state_signature>{identifier} { RET_VAL(PP_IDENTIFIER);                             }
 <define_state_signature>,            { RET_VAL(((int)*yytext));                            }
 <define_state_signature>")"          { BEGIN(define_state_definition); _definition.clear(); g_definition.Clear(); RET_VAL(((int)*yytext));}
 <define_state_signature>.            {}
+
 <define_state_definition>"/*"        { return_to_state = define_state_definition; BEGIN(define_generic_c_comment);}
 <define_state_definition>"//"        { 
 	BEGIN(define_generic_cpp_comment);
@@ -140,7 +155,9 @@
 <define_state_definition>\t          { _definition += " ";}
 <define_state_definition>\r          {}
 <define_state_definition>.           { _definition += yytext;}
+
 .                                    {}
+
 <incl>\n                             {BEGIN(INITIAL);}
 <incl>.                              {}
 
Index: pp.y
===================================================================
--- pp.y	(revision 4529)
+++ pp.y	(working copy)
@@ -51,6 +51,9 @@
 macros:   define_simple_macros
         | define_func_like_macros
         | if_cplusplus
+		| if_simple_macro
+		| ifdef_simple_macro
+		| ifndef_simple_macro
         | error {
             //wxPrintf(wxT("CodeLite: syntax error, unexpected token '%s' found\n"), pp_lval.c_str());
         }
@@ -110,5 +113,23 @@
         | args_list ',' PP_IDENTIFIER   { $$ = $1 + $2 + $3; }
         ;
 
+if_simple_macro: PP_IF PP_IDENTIFIER
+		{
+			PPTable::Instance()->AddUsed($2);
+		}
+		;
+
+ifdef_simple_macro: PP_IFDEF PP_IDENTIFIER
+		{
+			PPTable::Instance()->AddUsed($2);
+		}
+		;
+
+ifndef_simple_macro: PP_IFNDEF PP_IDENTIFIER
+		{
+			PPTable::Instance()->AddUsed($2);
+		}
+		;
+
 %%
 
Index: pptable.cpp
===================================================================
--- pptable.cpp	(revision 4529)
+++ pptable.cpp	(working copy)
@@ -456,6 +456,15 @@
 	}
 }
 
+void PPTable::AddUsed(wxString name)
+{
+	if(name.IsEmpty()) {
+		return;
+	}
+	name.Trim().Trim(false);
+	m_namesUsed.insert(name);
+}
+
 void PPTable::Print(wxFFile &fp)
 {
 	std::map<wxString, PPToken>::iterator iter = m_table.begin();
@@ -513,6 +522,11 @@
 	m_table.clear();
 }
 
+void PPTable::ClearNamesUsed()
+{
+	m_namesUsed.clear();
+}
+
 bool CLReplacePattern(const wxString& in, const wxString& pattern, const wxString& replaceWith, wxString &outStr)
 {
 	int where = pattern.Find(wxT("%0"));
Index: pptable.h
===================================================================
--- pptable.h	(revision 4529)
+++ pptable.h	(working copy)
@@ -7,6 +7,7 @@
 #include <vector>
 #include <string>
 #include <list>
+#include <set>
 
 struct CLReplacement {
 	bool                        is_compound;
@@ -81,6 +82,7 @@
 
 	static PPTable*             ms_instance;
 	std::map<wxString, PPToken> m_table;
+	std::set<wxString>          m_namesUsed;
 
 public:
 	static PPTable* Instance();
@@ -94,13 +96,19 @@
 	PPToken   Token(const wxString &name);
 	bool      Contains(const wxString &name);
 	void      Add  (const PPToken& token);
+	void      AddUsed(wxString name);
 	void      Print(wxFFile &fp);
 	wxString  Export();
 	void      Clear();
+	void      ClearNamesUsed();
 	void      Squeeze();
 
 	const std::map<wxString, PPToken>& GetTable() const {
 		return m_table;
 	}
+	
+	const std::set<wxString> GetNamesUsed() const {
+		return m_namesUsed;
+	}
 };
 #endif // PPTABLE_H

Is it a good start ? It gives me correct macro uses for the following file :

Code: Select all

#include <vector>
using  namespace   boost::filesystem;
#include <list>
using     namespace std;
#include <string>
#include <map>
#include <stdio.h>
#include <stdlib.h>

namespace bf = boost::filesystem ;

#define ERAN
#ifdef  ERAN
#define MY_CHECK(x, y) {\
    if(x < y){\
        printf();\
    }else{\
        printf();\
    }
    
#endif

#ifdef TOTO
	// hop
	#ifndef POUET
		// rem
	#endif
#endif

#if FOO
	// 
#endif

gives

ERAN FOO POUET TOTO

I need to improve it in order ta manage complexe #ifxxx definitions.

[eranif]

need to improve it in order ta manage complexe #ifxxx definitions.

What do you mean by "complex" definitions?

Eran