Spidered pages

Avatar
Hassan
  • updated
  • Answered
Hello,
I am new to this community.

I am more interested in spider-ed pages and find if bots are finding or crawling pages which are not very imortant for me. At the moment I can only see upto 10,000. What I will need to do if I want to look for more? is tool robust enough? Please help
Replies 7
Avatar
0
Michael
The program doesn’t allow users to set more than 10000 items in a table as such tables become very large and not all browsers can show them. However, it is possible to edit the configuration files manually and change this value.

If you wish to set the limit for all the profiles (and you set it in Options > Report > Contents), you need to edit the Main.cfg file located in the Config subfolder of the WebLog Expert data folder (usually “C:\Program Files\WebLog Expert” under Windows XP/2003 or older and “C:\ProgramData\WebLog Expert” under Windows Vista/2008/7). You can edit the file using Notepad or similar editor, search for value 10000 and replace it with a larger value.

Please note that if you use Windows Vista/2008/7, the C:\ProgramData is hidden by default (unlike the Program Files folder), so you need to set the Explorer to show hidden files and folders to access this folder.
Avatar
0
Hassan
Hello Michael,
I tried to increase the table items but tool is still showing me 10,000.

Please have a look at attached image. Thanks
Avatar
0
Michael
Have you changed the Main.cfg file to create more than 10000 items? If so, you should check that the custom report contents in Profile Properties > Report is disabled so the number of items from the program options is used.

If it doesn't help, could you please send the Main.cfg file it to us at support@weblogexpert.com so we check it?
Avatar
0
Hassan
I have yes and I was specifically looking for Spidered Pages, whcih is still showing 10,000.

But I can see above 14,000 errors 404 (Page Not Found) Errors
Avatar
0
Michael
Could you please send the Main.cfg file to us at support@weblogexpert.com so we check it?
Avatar
0
Hassan
Hello guys,
I closed the program and opend it again and that worked for me. sorry for being pain .. you have been very very helpful.

I am able to get just 1 report Spidered pages and list of 100000 iteams. I was thinking if I generate more iteams/tables and export csv. currently it crashes .. do you have any suggestions on minimum system requirements for 200,000 table for just "spidered pages" report?

Many thanks
Avatar
0
Michael
It seems that the program crashes because it requires too much memory to create the report. The program cannot use more than 2GB of RAM so I can only suggest you to create separate tables/subtables in separate reports. E.g. if you create a table on pages requested by spiders, you can create a separate report for pages requested by Googlebot, separate report for Bing Robot, etc. You can do it by using the "Spider" filter in the profile properties.