Preprocessing and scintilla_22

Discussion about CodeLite development process and patches
User avatar
eranif
CodeLite Plugin
Posts: 6056
Joined: Wed Feb 06, 2008 9:29 pm
Genuine User: Yes
IDE Question: C++
Contact:

Re: Preprocessing and scintilla_22

Post by eranif » Wed Nov 10, 2010 11:05 pm

jfouche wrote:Why did you put an empty file name in MACROS table ?
Can really remember :shock:

Sure, go a head and change it
Eran
Make sure you have read the HOW TO POST thread

jfouche
CodeLite Guru
Posts: 351
Joined: Mon Oct 20, 2008 7:26 pm
Genuine User: Yes
IDE Question: C++
Location: France
Contact:

Re: Preprocessing and scintilla_22

Post by jfouche » Wed Nov 10, 2010 11:41 pm

Well, I know : because there is no way to retrieve it currently ;)
I need to change the PPToken token class in order to add this member. No problem, I'll do it.
Jérémie

jfouche
CodeLite Guru
Posts: 351
Joined: Mon Oct 20, 2008 7:26 pm
Genuine User: Yes
IDE Question: C++
Location: France
Contact:

Re: Preprocessing and scintilla_22

Post by jfouche » Sat Nov 13, 2010 2:40 pm

hello Eran

Here are some news :
That works :D ... but I encounter some bad behaviors :
1 - The current file crawler just returns the include files from the parser include files, not from the workspace. So how can I PPScan a workspace file which is included by the editing one :

Code: Select all

#include "test.h" // not parsed, because not found by file crawler
#include <winnt.h> // OK
2 - As the file formating is not the same in the database, and in the string representing those files in CL, I did not manage the file name in the query. The pb is that if a macro is defined out of the included files, it will be seen as defined.

3 - The problems from this post need huge design changes in order to make it works correctly. but we will talk about this later.

I'll post a patch this afternoon.
Jérémie

jfouche
CodeLite Guru
Posts: 351
Joined: Mon Oct 20, 2008 7:26 pm
Genuine User: Yes
IDE Question: C++
Location: France
Contact:

Re: Preprocessing and scintilla_22

Post by jfouche » Sat Nov 13, 2010 6:58 pm

Here it is. Your feedback is welcome. It's not finished yet, but this step seems good to commit.
You do not have the required permissions to view the files attached to this post.
Jérémie

User avatar
eranif
CodeLite Plugin
Posts: 6056
Joined: Wed Feb 06, 2008 9:29 pm
Genuine User: Yes
IDE Question: C++
Contact:

Re: Preprocessing and scintilla_22

Post by eranif » Sat Nov 13, 2010 8:31 pm

Some comments:
- I think you should return the output as a class and place it in the SetClientData of the command event (e.g. dont place the file name in the GetString() method of the wxCommandEvent)
- As part of the parser request you should also pass the workspace files list
have a look at Manager::RetagWorkspace():

Code: Select all

wxArrayString projects;
	GetProjectList ( projects );

	std::vector<wxFileName> projectFiles;
	for ( size_t i=0; i<projects.GetCount(); i++ ) {
		ProjectPtr proj = GetProject ( projects.Item ( i ) );
		if ( proj ) {
			proj->GetFiles( projectFiles, true );
		}
	}

	// Create a parsing request
	ParseRequest *parsingRequest = new ParseRequest();
	for (size_t i=0; i<projectFiles.size(); i++) {
		// filter any non valid coding file
		if(!TagsManagerST::Get()->IsValidCtagsFile(projectFiles.at(i)))
			continue;
		parsingRequest->_workspaceFiles.push_back( projectFiles.at(i).GetFullPath().mb_str(wxConvUTF8).data() );
	}
- in ParseThread::ProcessInterrestingMacros(ParseRequest *req), the call to "macros.RemoveLast()" can cause to segault, if "macros" is empty

Eran
Make sure you have read the HOW TO POST thread

jfouche
CodeLite Guru
Posts: 351
Joined: Mon Oct 20, 2008 7:26 pm
Genuine User: Yes
IDE Question: C++
Location: France
Contact:

Re: Preprocessing and scintilla_22

Post by jfouche » Sun Nov 14, 2010 10:40 pm

Here it is. Once again, it's not perfect, due to the fact we look to all macros, even of the current file (so it's all grayed in a header file, when there is guard block).
I really need a way to find in specific files in the MACROS database.
You do not have the required permissions to view the files attached to this post.
Jérémie

User avatar
eranif
CodeLite Plugin
Posts: 6056
Joined: Wed Feb 06, 2008 9:29 pm
Genuine User: Yes
IDE Question: C++
Contact:

Re: Preprocessing and scintilla_22

Post by eranif » Fri Nov 19, 2010 11:01 pm

Hi Jeremie,

I finally got the time to apply this patch, and here are my comments:

- When I first ran the patached codelite it did not work, I then realized that I need to perform a full retag of codelite so the "Macros" table will be populated. This should be done automatically (codelite has a mechanism for schema changes) - this can be achieved by changing the value of "gTagsDatabaseVersion" global variable in tags_storage_sqlite3.h, when modified codelite will retag the workspace and will update the schema version
- The method 'Manager::UpdatePreprocessorFile' is called twice once from the event 'OnPageChanged' and once from ReloadFile() from cl_editor.cpp
- But the biggest problem is that 'Manager::UpdatePreprocessorFile' is *very* slow. I tried it on codelite workspace and it took several seconds for a file to load, but since we also call it from 'OnPageChanged' switching tabs also become laggy (2-3 seconds to swtich a tab...)

The cause of the laggy behvior is the code that collects the 'workspace files' and filter them - I will see how this code can be optimized
- And last problem is the one you mentioned: block guards cause the entire file to gray out

Improvements:
- We need to provide means in the UI to disable this option
- I am going to add 'C/C++ Parser options' to the project level. Once added, I am going to add two options:
** Automatically detect macros
** User macros - this way the user can set a macros in case the automatic process fails to detect it correctly
In addition, I will add an option to set 'search paths' per build configuration

Eran
Make sure you have read the HOW TO POST thread

User avatar
eranif
CodeLite Plugin
Posts: 6056
Joined: Wed Feb 06, 2008 9:29 pm
Genuine User: Yes
IDE Question: C++
Contact:

Re: Preprocessing and scintilla_22

Post by eranif » Fri Nov 19, 2010 11:35 pm

Update:
eranif wrote:- But the biggest problem is that 'Manager::UpdatePreprocessorFile' is *very* slow. I tried it on codelite workspace and it took several seconds for a file to load, but since we also call it from 'OnPageChanged' switching tabs also become laggy (2-3 seconds to swtich a tab...)
I managed to reduce the overhead of loading the file, the main slow down was this line:

In Manager::UpdatePreprocessorFile:

Code: Select all

if (projFileName == filename) {
wxFileName == wxFileName is *very* slow. Changing this to comparison between the full paths of the two reduced 99% of the slowdown.
Still, we need to optimize the gathering the workspace files code.

Also, I removed the call to Manager::UpdatePreprocessorFile() from LEditor::ReloadFile() (it seems that handling it from Mainbook::OnPageChanged is enough)
I will commit the changes once I will add a GUI option to disable this functionality

Eran
Make sure you have read the HOW TO POST thread

jfouche
CodeLite Guru
Posts: 351
Joined: Mon Oct 20, 2008 7:26 pm
Genuine User: Yes
IDE Question: C++
Location: France
Contact:

Re: Preprocessing and scintilla_22

Post by jfouche » Sat Nov 20, 2010 1:27 am

Hi Eran

Glad to see your review about my patch. It's very interesting.
A pause was also good for me to think about this improvement. Here are my thought :

The main question I was asking myself is the following : What are really the macros value used by a file ?
The answer is not so easy : We need to work as a compiler : include the headers in the real order, and manage the macro retrieving / storing during this job. And of course, it depends on the current workspace configuration (think about the DEBUG macro for example). Retrieving headers as a std::set change the order (moreover, IIRC, CL doesn't parse directly an header when it find one).

The second point is to store the macros value even for simple macros for #if lines :

Code: Select all

#if MACRO == 7
Well, I suppose it needs lot's of changes...

Maybe clang could help us for this task (when you'll integrate to CL ;) )
Jérémie

jfouche
CodeLite Guru
Posts: 351
Joined: Mon Oct 20, 2008 7:26 pm
Genuine User: Yes
IDE Question: C++
Location: France
Contact:

Re: Preprocessing and scintilla_22

Post by jfouche » Mon Nov 22, 2010 6:20 pm

Hi Eran

Could you tell me why the file crawler returns files with '/' instead of '\' under Windows ? May I change this ? and How ?
Now, I got something cool : It's faster and better (even for guard block in headers).
Jérémie

Post Reply