Database computation
Is it true that if any of the log files in the corpus changes, a complete recalculation of database is needed? What happens if the calculation does not complete before the next scheduled caclculation?
Replies 4
Yes, it's true. The goal of the program is to create a report that reflects current state of log files. If it detects that any log file has changed, it needs to reanalyze all logs as it cannot exclude data from old log and include data from new log. The only exception are usual text uncompressed logs with appended lines - in most cases the program should detect that there were lines added and just analyze these lines without reanalyzing all data.
If you use the built-in scheduler, it always performs analysis of one profile at a time only. So if a calculation does not complete before the new scheduled one, the second calculation is started only after the first one finishes.
If you use the built-in scheduler, it always performs analysis of one profile at a time only. So if a calculation does not complete before the new scheduled one, the second calculation is started only after the first one finishes.
It seems to me that you might want to adjust the architecture so that the log files can be completely discarded at the end of each reporting period. Even zipped, those ancient files (1-2 years old) can waste space..