Preprocessing and scintilla_22

[eranif]

Why did you put an empty file name in MACROS table ?

Can really remember

Sure, go a head and change it
Eran

[jfouche]

Well, I know : because there is no way to retrieve it currently
I need to change the PPToken token class in order to add this member. No problem, I'll do it.

[jfouche]

hello Eran

Here are some news :
That works ... but I encounter some bad behaviors :
1 - The current file crawler just returns the include files from the parser include files, not from the workspace. So how can I PPScan a workspace file which is included by the editing one :

Code: Select all

#include "test.h" // not parsed, because not found by file crawler
#include <winnt.h> // OK

2 - As the file formating is not the same in the database, and in the string representing those files in CL, I did not manage the file name in the query. The pb is that if a macro is defined out of the included files, it will be seen as defined.

3 - The problems from this post need huge design changes in order to make it works correctly. but we will talk about this later.

I'll post a patch this afternoon.

[jfouche]

Here it is. Your feedback is welcome. It's not finished yet, but this step seems good to commit.

[eranif]

Some comments:
- I think you should return the output as a class and place it in the SetClientData of the command event (e.g. dont place the file name in the GetString() method of the wxCommandEvent)
- As part of the parser request you should also pass the workspace files list
have a look at Manager::RetagWorkspace():

Code: Select all

wxArrayString projects;
	GetProjectList ( projects );

	std::vector<wxFileName> projectFiles;
	for ( size_t i=0; i<projects.GetCount(); i++ ) {
		ProjectPtr proj = GetProject ( projects.Item ( i ) );
		if ( proj ) {
			proj->GetFiles( projectFiles, true );
		}
	}

	// Create a parsing request
	ParseRequest *parsingRequest = new ParseRequest();
	for (size_t i=0; i<projectFiles.size(); i++) {
		// filter any non valid coding file
		if(!TagsManagerST::Get()->IsValidCtagsFile(projectFiles.at(i)))
			continue;
		parsingRequest->_workspaceFiles.push_back( projectFiles.at(i).GetFullPath().mb_str(wxConvUTF8).data() );
	}

- in ParseThread::ProcessInterrestingMacros(ParseRequest *req), the call to "macros.RemoveLast()" can cause to segault, if "macros" is empty

Eran

[jfouche]

Here it is. Once again, it's not perfect, due to the fact we look to all macros, even of the current file (so it's all grayed in a header file, when there is guard block).
I really need a way to find in specific files in the MACROS database.

[eranif]

Hi Jeremie,

I finally got the time to apply this patch, and here are my comments:

- When I first ran the patached codelite it did not work, I then realized that I need to perform a full retag of codelite so the "Macros" table will be populated. This should be done automatically (codelite has a mechanism for schema changes) - this can be achieved by changing the value of "gTagsDatabaseVersion" global variable in tags_storage_sqlite3.h, when modified codelite will retag the workspace and will update the schema version
- The method 'Manager::UpdatePreprocessorFile' is called twice once from the event 'OnPageChanged' and once from ReloadFile() from cl_editor.cpp
- But the biggest problem is that 'Manager::UpdatePreprocessorFile' is *very* slow. I tried it on codelite workspace and it took several seconds for a file to load, but since we also call it from 'OnPageChanged' switching tabs also become laggy (2-3 seconds to swtich a tab...)

The cause of the laggy behvior is the code that collects the 'workspace files' and filter them - I will see how this code can be optimized
- And last problem is the one you mentioned: block guards cause the entire file to gray out

Improvements:
- We need to provide means in the UI to disable this option
- I am going to add 'C/C++ Parser options' to the project level. Once added, I am going to add two options:
** Automatically detect macros
** User macros - this way the user can set a macros in case the automatic process fails to detect it correctly
In addition, I will add an option to set 'search paths' per build configuration

Eran

[eranif]

Update:

- But the biggest problem is that 'Manager::UpdatePreprocessorFile' is *very* slow. I tried it on codelite workspace and it took several seconds for a file to load, but since we also call it from 'OnPageChanged' switching tabs also become laggy (2-3 seconds to swtich a tab...)

I managed to reduce the overhead of loading the file, the main slow down was this line:

In Manager::UpdatePreprocessorFile:

Code: Select all

if (projFileName == filename) {

wxFileName == wxFileName is *very* slow. Changing this to comparison between the full paths of the two reduced 99% of the slowdown.
Still, we need to optimize the gathering the workspace files code.

Also, I removed the call to Manager::UpdatePreprocessorFile() from LEditor::ReloadFile() (it seems that handling it from Mainbook::OnPageChanged is enough)
I will commit the changes once I will add a GUI option to disable this functionality

Eran

[jfouche]

Hi Eran

Glad to see your review about my patch. It's very interesting.
A pause was also good for me to think about this improvement. Here are my thought :

The main question I was asking myself is the following : What are really the macros value used by a file ?
The answer is not so easy : We need to work as a compiler : include the headers in the real order, and manage the macro retrieving / storing during this job. And of course, it depends on the current workspace configuration (think about the DEBUG macro for example). Retrieving headers as a std::set change the order (moreover, IIRC, CL doesn't parse directly an header when it find one).

The second point is to store the macros value even for simple macros for #if lines :

Code: Select all

#if MACRO == 7

Well, I suppose it needs lot's of changes...

Maybe clang could help us for this task (when you'll integrate to CL )

[jfouche]

Hi Eran

Could you tell me why the file crawler returns files with '/' instead of '\' under Windows ? May I change this ? and How ?
Now, I got something cool : It's faster and better (even for guard block in headers).