Weblog Expert speed issue

Morgan Storey

9 years ago
updated 9 years ago
Answered

I currently have 330gb of IISlogs (they are compressed with folder compression down to 90gb) from multiple sites that I have set to analyze with Weblog expert, I have got DNS resolution enabled (just bumped this to 2000 threads). When this was sub 200gb it took much less than 12hours to complete, now though it is taking more than 24hours.
I have my Schedule set to simply analyse all profiles at 7pm (priority High), this runs a pre-script that downloads all the logs from remote servers, logs when this script starts and finishes, and then runs a post script so I know when processing has completed;

From when it was sub 200gb;
Download started Thu 04/09/2014 22:00:00.84
Download Complete Thu 04/09/2014 22:00:01.20
Processing complete Fri 05/09/2014 6:54:06.02

More recent (330gb), I'd moved the start time earlier to allow for the increase in logs;
Download started Sun 10/05/2015 19:00:00.45
0 instances of sftpc running
Download Complete Sun 10/05/2015 19:06:57.87
Processing complete Tue 12/05/2015 0:41:13.32

If I log in while the analysis is running I can see WLExpert.exe running as system, using usually 0% CPU and a few hundred MB of ram. In perfmon I can see disk Read is not super high at around 24MBps and queues average around 0.2.

If I split up the analysis will it run them simultaneously? Is there something wrong with this that is causing the low CPU utilization? Could it simply be the windows folder compression causing issues?

Replies 6

Oldest first
Newest first Oldest first

9 years ago

Is there anyway to set the analysis so that it just does logs that have changed since the last analysis?

Reply Link

Michael

The problem is probably caused by the DNS lookups, we don't recommend to use it for such large log files as it works slowly and can cause network problems or work unreliably if number of lookup threads is very high (like in your case).

Usually the program uses a lot of CPU, as it runs analysis using multiple threads. So you should try to run the analysis without the DNS lookup enabled and check if it works correctly in your case.

I don't think that folder compression causes the issue, but it most cases it is better to compress logs to .zip or .gz files one log per archive. Zipped logs are usually 10-20 times smaller than uncompressed ones.

In most cases the program doesn't need to reanalyze already analyzed logs, but it depends on how log files are managed. If new logs are added, there shouldn't be problems. However, if some logs are renamed, the program will need to reanalyze the logs to create reports that reflect the currently available logs.

The best solution is to create a folder with compressed logs and place daily compressed logs to it. In this case WebLog Expert should be set to analyze this archive folder, not the original log folder. In such case logs will be smaller and WebLog Expert won't have problems with changed logs, as today's logs won't be stored in this folder.

I don't change any of the older logs, just add new ones.
I need the DNS lookups. Will more threads be faster or slower for this?

So are you saying if say for example I had a folder like the below on Day 1, and analysed the logs;

W3SVC1234567
10/05/2015 11:59 PM 187,099,768 u_ex150510.log
11/05/2015 11:59 PM 250,847,173 u_ex150511.log
13/05/2015 12:00 AM 250,704,316 u_ex150512.log
14/05/2015 12:00 AM 255,863,772 u_ex150513.log

then on day 2 I analyse the logs again, but move out all the logs from day 1 so I just have the below;
14/05/2015 07:00 PM 223,521,701 u_ex150514.log

That the weblog analysis would simply update with the new log file?

Optimal number of DNS threads on the computer and network configuration. The default value is 50 threads, but you should probably try 100 or 200 threads. 2000 threads is probably too much, as it can cause problems at the computer or DNS server (what is actually did).

Regarding the logs - you shouldn't delete logs from the folder that is analyzed by the program. You should only add new logs, and it's better to add logs that won't be changed anymore.

I did a test and deleted out the newest two logs, ran an analysis, and then moved all already analysed logs out and moved the two newest in, and the total hits dropped, as though it had only analysed the logs that were in the folder.

I am deleting old logs (older than 365 days), are you saying I shouldn't delete them at all and that will fix it, from my testing that doesn't seem right.

I have dropped the DNS lookups from my test of 2000 to 400 and will see how long it takes tonight with the main task.

The program always produces reports based on logs that exist in the analyzed folder only. However, if the program is set to analyze logs for all time range ("All activity") and you just add new files to the folder, the program shouldn't reanalyze all logs and should just analyze the new logs.