Preprocessing and scintilla_22

Discussion about CodeLite development process and patches
User avatar
eranif
CodeLite Plugin
Posts: 6056
Joined: Wed Feb 06, 2008 9:29 pm
Genuine User: Yes
IDE Question: C++
Contact:

Re: Preprocessing and scintilla_22

Post by eranif » Tue Nov 02, 2010 11:55 am

Indeed. You are correct.

But I guess this can be done in the 'OnSave'

Eran
Make sure you have read the HOW TO POST thread

jfouche
CodeLite Guru
Posts: 351
Joined: Mon Oct 20, 2008 7:26 pm
Genuine User: Yes
IDE Question: C++
Location: France
Contact:

Re: Preprocessing and scintilla_22

Post by jfouche » Sat Nov 06, 2010 4:55 pm

Hello Eran

Back again :)

Some questions :
1 - Do you know how I can add comments in lex file, in order to help myself. I would like to add some like that :

Code: Select all

// #if ...
//     ^
<if_state>... {}
2 - Do you know how to handle multiple MACROS in one line using yacc. For example, parsing the following :

Code: Select all

#if defined(__WXMSW__) && defined(__X__)

Code: Select all

if_macros: PP_IFDEF PP_IDENTIFIER * // the * may say multiple MACROS
		{
			PPTable::Instance()->AddUsed($2);
			// What Can I use here to handle all other macro use
		}
		;
Jérémie

User avatar
eranif
CodeLite Plugin
Posts: 6056
Joined: Wed Feb 06, 2008 9:29 pm
Genuine User: Yes
IDE Question: C++
Contact:

Re: Preprocessing and scintilla_22

Post by eranif » Sat Nov 06, 2010 9:25 pm

To handle a multiple 'defined' you need to create a recursive rule.

For example:

Code: Select all

#if defined(__WXMSW__) && defined(__X__)
You can create a rule like:

Code: Select all

if_def: TOKEN_IF sequence_of_conditions
        ;

sequence_of_conditions: TOKEN_DEFINED '(' TOKEN_CONDITION ')'                                  { printf("Found condition %s\n", $4.c_str()); }
        | sequence_of_conditions logical_operator TOKEN_DEFINED '(' TOKEN_CONDITION ')' { printf("Found condition %s\n", $5.c_str()); }
        ;

logical_operator: TOKEN_AND
                       | TOKEN_OR
                       ;
this recursive rule will print both conditions __WXMSW__ and __X__ (and any other conditions as long they are 'concatenated' with either && or || 'logical_operator')

HTH,
Eran
Make sure you have read the HOW TO POST thread

jfouche
CodeLite Guru
Posts: 351
Joined: Mon Oct 20, 2008 7:26 pm
Genuine User: Yes
IDE Question: C++
Location: France
Contact:

Re: Preprocessing and scintilla_22

Post by jfouche » Sat Nov 06, 2010 11:48 pm

Thanks a lot for the tips

Here is a patch to apply in the PreProcessor/filecrawler directory. It can find all used macros (see test.h), but one :shock:

Code: Select all

#if E 
	// 
#endif
I used this :

Code: Select all

if_macros: PP_IF if_sequence
        ;

elif_macros: PP_ELIF if_sequence
		;

if_sequence: if_condition
		| if_sequence logical_operator if_condition
		;
		
if_condition: PP_IDENTIFIER
		{
			// DO NOT WORK ! WHY ?
			wxPrintf(wxT("simple : %s\n"), $1.c_str());
			PPTable::Instance()->AddUsed($1);
		}
		| PP_IDENTIFIER test_operator PP_INT
		{
			wxPrintf(wxT("test : %s\n"), $1.c_str());
			PPTable::Instance()->AddUsed($1);
		}
		| PP_DEFINED '(' PP_IDENTIFIER ')' 
		{
			PPTable::Instance()->AddUsed($3);
		}
        ;
What did I miss ?

Moreover, may I ask you to commit this patch (or the one you may fixe) in SCINTILLA_22 branch ? Or could you give me access to SVN ? Because I think I will not be able to do all the job alone, and I'll do it step by step.
Thanks
You do not have the required permissions to view the files attached to this post.
Jérémie

User avatar
eranif
CodeLite Plugin
Posts: 6056
Joined: Wed Feb 06, 2008 9:29 pm
Genuine User: Yes
IDE Question: C++
Contact:

Re: Preprocessing and scintilla_22

Post by eranif » Mon Nov 08, 2010 1:47 pm

Hi,

I committed the patch with modifications:

- The grammar now collects all identifiers used with the #if pre-processor (running the test, it will now prints E-O)
- Updated the grammar to collect "complex" conditions such as:

Code: Select all

#if defined(M) && N=7
or any various or N conditions
- Added the PP_LOWERTHAN / PP_GREATERTHAN tokens (we need to extend it for cover the other operators as well, like '>=' '<=' etc)
jfouche wrote:What did I miss ?
Apparently, adding rule to handle the '#endif' fixed this

Code: Select all

end_if : PP_ENDIF
	   ;
Some tips:
- I added a new method that can be used from the main.cpp file: testTokens() - this will print the macros ID as seen by the lexer without calling the parser itself (useful when you want to make sure that a token was passed to the parser)
- In the main rule, (i.e. 'macros:' ) there is a commented line, uncomment it and it will usually will tell you where the parser failed.

Eran
Make sure you have read the HOW TO POST thread

jfouche
CodeLite Guru
Posts: 351
Joined: Mon Oct 20, 2008 7:26 pm
Genuine User: Yes
IDE Question: C++
Location: France
Contact:

Re: Preprocessing and scintilla_22

Post by jfouche » Mon Nov 08, 2010 5:19 pm

Hi Eran

Let's go to step 2 : Update Database
I suppose we can use the existing MACROS table, to store all defined macros (even the empty ones). But it may decrease performances while finding macro replacement (as this table will probably greatly increase). What is your opinion : use existing table, or create a new one (simpler) :

Code: Select all

create table if not exists SIMPLE_MACROS (ID INTEGER PRIMARY KEY AUTOINCREMENT, file string, line integer, name string)
Jérémie

User avatar
eranif
CodeLite Plugin
Posts: 6056
Joined: Wed Feb 06, 2008 9:29 pm
Genuine User: Yes
IDE Question: C++
Contact:

Re: Preprocessing and scintilla_22

Post by eranif » Mon Nov 08, 2010 7:48 pm

jfouche wrote:What is your opinion : use existing table, or create a new one (simpler) :
My opinion is to use a separate table, not only because I fear for completion, but also because I I think that adding a new table will reduce the chance of breaking something else

Also, make sure to define search indexes (probably on the 'file' column) since the most common query on this table will be:

Code: Select all

select * from simple_macros where file='...'
Eran
Make sure you have read the HOW TO POST thread

jfouche
CodeLite Guru
Posts: 351
Joined: Mon Oct 20, 2008 7:26 pm
Genuine User: Yes
IDE Question: C++
Location: France
Contact:

Re: Preprocessing and scintilla_22

Post by jfouche » Wed Nov 10, 2010 12:28 am

I'm currently modifying the ITagStorage interface in order to add a way to retrieve all macros defined by some include files :

Code: Select all

class ITagsStorage
{
	...
	/**
	 * @brief return the macros defined by some files
	 * @param files The files which define the macros we are looking for
	 * @param macros [out] The defined macros
	 */
	virtual void GetMacrosDefined(const wxArrayString& files, wxArrayString& macros) = 0;
};
Do you agree with this signature ? It will find inside the new SIMPLE_MACRO table (which only contains simple macros, eg : which doesn't expand to something). I'll probably use the existing StoreMacro method to fill this this new table (if the replacement is empty, store the macro in the SIMPLE_MACRO table).
What do you think ?
Jérémie

User avatar
eranif
CodeLite Plugin
Posts: 6056
Joined: Wed Feb 06, 2008 9:29 pm
Genuine User: Yes
IDE Question: C++
Contact:

Re: Preprocessing and scintilla_22

Post by eranif » Wed Nov 10, 2010 1:02 am

Sure, looks fine to me.

Looking for the patch.

Note: I merged all the branch content into the trunk, so the branch is no longer needed

Eran
Make sure you have read the HOW TO POST thread

jfouche
CodeLite Guru
Posts: 351
Joined: Mon Oct 20, 2008 7:26 pm
Genuine User: Yes
IDE Question: C++
Location: France
Contact:

Re: Preprocessing and scintilla_22

Post by jfouche » Wed Nov 10, 2010 10:59 pm

Why did you put an empty file name in MACROS table ? I suppose it's for speed.
May I change it to the real one ?
Jérémie

Post Reply