Issues on large log sets

Avatar
vesuvious
  • Answered
We recently purchased WebLog Expert Enterprise to test as a replacement for an older Log Analyzer and it has been working great for most of our sites. We are running the software on a 2008 R2 server (x64) which will pull the website logs from different shares and dump the reports to an IIS hosted directory. We have run into an issue where larger sites (10 GB of Logs , 4 redundant site locations = 40GB) are causing the program to crash during the initial analyze run. We would originally get memory errors on these sites (4 GB RAM) but we dropped in more memory (12 GB total) and now the program just crashes completely after running for a few hours. Do you know if we still may need more memory or if this is a software bug? Eventually we would love to use this software for all of our sites if we can get everything to run correctly.
Replies 9
Avatar
1
Michael
If I understand correctly, you create static HTML reports with the "Create HTML reports for additional time ranges" setting enabled in Profile Properties > Time Range. In this case I recommend you to create dynamic HTML reports instead. These reports are shown by web server that shows them using information from the database.

Dynamic HTML reports allow you to choose any time range when you view them and also has additional capabilities like trends for individual items. You can view a report sample at http://demo.weblogexpert.com/Login.as...

These reports also don't have such requirements for memory as static HTML reports. Even if an individual table requires too much memory to be generated, the server will just show a suggestion to choose a smaller time range, while static HTML reports won't be generated at all.

You can find more information on dynamic HTML reports at http://www.weblogexpert.com/help/wlex...

Unfortunately it's not possible to uncheck some reports without reanalyzing log files.

We considered creating a 64-bit version of the program but there are some technical difficulties, mostly because of some third-party libraries we use. However, probably we'll create a 64-bit version later, but I don't think that we'll do it in the near versions of the program.
Avatar
0
vesuvious
With the current stable version of Weblog Expert you can only have dynamic reports by using the internal web server. I did see that in the "Beta 2" there is the ability to host dynamic reports in IIS, but I cannot seem to figure out how to enable this function.

Unfortunately without single sign on or at least the ability to turn authentication off, we cannot use the built in web server and instead are dumping the reports in an IIS hosted directory and creating the front end ourselves.

I was able to get larger logs to work by turning off some reports, and if the reports are necessary we may need to just do one site at a time instead of combining all of the logs from each redundant site.

The software all together is fantastic and I look forward to future versions that may resolve some of these minor issues. Thanks again for all of your help.
Avatar
0
Michael
If you wish to switch to IIS, you need to open the Web Server dialog and click the "Change..." button to the left of the URL label. Probably we should make it clearer that this button also allows to change the server mode, not just change the URL.

It is possible to specify username and password in the URL so you won't need to enter them manually, for example:
http://demo.weblogexpert.com/Login.aspx?ReturnUrl=/Report.aspx?id=1&username=demo&password=demo
Will it suit your needs?
Avatar
0
vesuvious
Thanks Michael that will work well for us.

One more thing:

I attempted to patch the application using this method:

http://maketecheasier.com/increase-me...

which has worked before for me to allow 32-bit applications to utilize up to 4GB of memory. It seems that the software monitors memory usage and quits once it gets close to 2GB in order to avoid errors so this patching will not work. I Just wanted to make sure this is the case and it is not something I can attempt to work around.
Avatar
0
Michael
Yes, the program monitors memory usage to avoid crashing so it won't use more than 2GB in any case.
Avatar
0
Michael
It shouldn't be a memory problem as the program cannot use more than 2GB of RAM anyway. However, you should check that there is enough disk space available for the program to store the database.

Could you please send more information on the crash including exception code, exception offset and other information provided by Windows?

I also recommend you to try to copy the logs to a local computer and try to analyze them, maybe the problem is caused by some kind of connection issue.

BTW is 40GB the size of compressed or uncompressed log files?
Avatar
0
vesuvious
40GB is the size of a mix of both compressed and uncompressed log files. The files are compressed monthly. The average size of each uncompressed log for these websites is around 250MB.

There is plenty of space for the cache databases, the server has a few TB's and is dedicated only to processing these statistics.

The issue only occurs during the report creation phase so I do not believe connection to be a problem since the cache database is stored locally.

I will add more information on the exception when it next occurs, should be in a few hours since I am retrying another large file set.

I am also getting errors about not having enough memory to create some reports, is it possible to allow the program to utilize more than 2GB so that it can complete these tasks? Like I said this server is dedicated to this process so I am not worried about resource issues.
Avatar
0
Michael
Do you create PDF or HTML reports?

The program is a 32-bit application so unfortunately it cannot use more than 2GB.
Avatar
0
vesuvious
Thank you for your response. I was unable to get the program to crash again so maybe that was just a specific report causing that issue.

I am creating HTML reports for multiple time-frames.

I understand the limitations on 32-bit programs so I am fine with turning off some of the reports that use more memory for now. Is it possible to make changes to the report without having to re-analyze the entire site? Its a bit rough to fix issues when I have to make one change and then wait 5-6 hours to see if it worked before I make another.

Also, do you believe a 64-bit version will be available in the future?