Preprocessing and scintilla_22

[eranif]

Indeed. You are correct.

But I guess this can be done in the 'OnSave'

Eran

[jfouche]

Hello Eran

Back again

Some questions :
1 - Do you know how I can add comments in lex file, in order to help myself. I would like to add some like that :

Code: Select all

// #if ...
//     ^
<if_state>... {}

2 - Do you know how to handle multiple MACROS in one line using yacc. For example, parsing the following :

Code: Select all

#if defined(__WXMSW__) && defined(__X__)

Code: Select all

if_macros: PP_IFDEF PP_IDENTIFIER * // the * may say multiple MACROS
		{
			PPTable::Instance()->AddUsed($2);
			// What Can I use here to handle all other macro use
		}
		;

[eranif]

To handle a multiple 'defined' you need to create a recursive rule.

For example:

Code: Select all

#if defined(__WXMSW__) && defined(__X__)

You can create a rule like:

Code: Select all

if_def: TOKEN_IF sequence_of_conditions
        ;

sequence_of_conditions: TOKEN_DEFINED '(' TOKEN_CONDITION ')'                                  { printf("Found condition %s\n", $4.c_str()); }
        | sequence_of_conditions logical_operator TOKEN_DEFINED '(' TOKEN_CONDITION ')' { printf("Found condition %s\n", $5.c_str()); }
        ;

logical_operator: TOKEN_AND
                       | TOKEN_OR
                       ;

this recursive rule will print both conditions __WXMSW__ and __X__ (and any other conditions as long they are 'concatenated' with either && or || 'logical_operator')

HTH,
Eran

[jfouche]

Thanks a lot for the tips

Here is a patch to apply in the PreProcessor/filecrawler directory. It can find all used macros (see test.h), but one

Code: Select all

#if E 
	// 
#endif

I used this :

Code: Select all

if_macros: PP_IF if_sequence
        ;

elif_macros: PP_ELIF if_sequence
		;

if_sequence: if_condition
		| if_sequence logical_operator if_condition
		;
		
if_condition: PP_IDENTIFIER
		{
			// DO NOT WORK ! WHY ?
			wxPrintf(wxT("simple : %s\n"), $1.c_str());
			PPTable::Instance()->AddUsed($1);
		}
		| PP_IDENTIFIER test_operator PP_INT
		{
			wxPrintf(wxT("test : %s\n"), $1.c_str());
			PPTable::Instance()->AddUsed($1);
		}
		| PP_DEFINED '(' PP_IDENTIFIER ')' 
		{
			PPTable::Instance()->AddUsed($3);
		}
        ;

What did I miss ?

Moreover, may I ask you to commit this patch (or the one you may fixe) in SCINTILLA_22 branch ? Or could you give me access to SVN ? Because I think I will not be able to do all the job alone, and I'll do it step by step.
Thanks

[eranif]

Hi,

I committed the patch with modifications:

- The grammar now collects all identifiers used with the #if pre-processor (running the test, it will now prints E-O)
- Updated the grammar to collect "complex" conditions such as:

Code: Select all

#if defined(M) && N=7

or any various or N conditions
- Added the PP_LOWERTHAN / PP_GREATERTHAN tokens (we need to extend it for cover the other operators as well, like '>=' '<=' etc)

What did I miss ?

Apparently, adding rule to handle the '#endif' fixed this

Code: Select all

end_if : PP_ENDIF
	   ;

Some tips:
- I added a new method that can be used from the main.cpp file: testTokens() - this will print the macros ID as seen by the lexer without calling the parser itself (useful when you want to make sure that a token was passed to the parser)
- In the main rule, (i.e. 'macros:' ) there is a commented line, uncomment it and it will usually will tell you where the parser failed.

Eran

[jfouche]

Hi Eran

Let's go to step 2 : Update Database
I suppose we can use the existing MACROS table, to store all defined macros (even the empty ones). But it may decrease performances while finding macro replacement (as this table will probably greatly increase). What is your opinion : use existing table, or create a new one (simpler) :

Code: Select all

create table if not exists SIMPLE_MACROS (ID INTEGER PRIMARY KEY AUTOINCREMENT, file string, line integer, name string)

[eranif]

What is your opinion : use existing table, or create a new one (simpler) :

My opinion is to use a separate table, not only because I fear for completion, but also because I I think that adding a new table will reduce the chance of breaking something else

Also, make sure to define search indexes (probably on the 'file' column) since the most common query on this table will be:

Code: Select all

select * from simple_macros where file='...'

Eran

[jfouche]

I'm currently modifying the ITagStorage interface in order to add a way to retrieve all macros defined by some include files :

Code: Select all

class ITagsStorage
{
	...
	/**
	 * @brief return the macros defined by some files
	 * @param files The files which define the macros we are looking for
	 * @param macros [out] The defined macros
	 */
	virtual void GetMacrosDefined(const wxArrayString& files, wxArrayString& macros) = 0;
};

Do you agree with this signature ? It will find inside the new SIMPLE_MACRO table (which only contains simple macros, eg : which doesn't expand to something). I'll probably use the existing StoreMacro method to fill this this new table (if the replacement is empty, store the macro in the SIMPLE_MACRO table).
What do you think ?

[eranif]

Sure, looks fine to me.

Looking for the patch.

Note: I merged all the branch content into the trunk, so the branch is no longer needed

Eran

[jfouche]

Why did you put an empty file name in MACROS table ? I suppose it's for speed.
May I change it to the real one ?