How can we get cron results?

[abtlang]

Hi, I've seen quite a few questions about a cron log, but I don't think I've ever seen any solution as to how to get the results of the cron job set up in version 2.0.1.

I have no problems with running the cron (check.php) and I've seen that it does indeed recognize broken links and reciprocal links that have been removed.

However, I've also seen that there doesn't seem to be any way to get a report on which links it has found to be broken, and on which reciprocal links have been removed.

Since I'm just setting up my directory, at the moment, with just 10 or so links, that's no problem, because I can go into the Admin Panel and see which link is now indicated as being not reciprocal. But, when I have hundreds of links, some of them reciprocal, some of them non-reciprocal, it's going to take me a long time to work out which links, if any, have changed status as a result of a cron job (I notice that if the cron job finds that a reciprocal link no longer exists, it simply marks the corresponding listing as non-reciprocal - but it doesn't tell you what it's done, and in fact it doesn't tell you whether it's done anything or not!).

Is there any way of having the cron send a report of the results? Or of having a log on the server that can be consulted?

Sorry if this is answered somewhere else - I've certainly spent a lot of time looking and I haven't found it!

Thanks in advance,

Terry.

[John Turner]

Hi abtlang and welcome.

I have implemented modification
After cronjob is finished
Email would be sent with results

open your '../cron/check.php'

***find following code

$query .= "OR (`last_check_date` IS NULL) ";
$query .= "ORDER BY `id` ASC LIMIT {$num_links}";
$links =& $gDb->getAll($query);

***add after it following code

$count_broken=0;
$count_reciprocal=0;
$count_pagerank=0;
$count_all=0;


***find following code

if (empty($headers))
{
$link_header = 1;
}
else
{
$link_header = (int)$headers['Status'];


***add after it following code

$count_broken++;


***find following code

$content = get_page_content($url);
if (!empty($content))
{
$recip_valid = (int)has_url($content, $settings['reciprocal_text']);

***add after it following code

$count_reciprocal++;

***find following code

$pr_content = trim(get_page_content($pr_url));
if (preg_match('/^Rank_\d{1}:\d{1}:(\d+)$/', $pr_content, $matches))
{
$pagerank = (int)$matches[1];


***add after it following code

$count_pagerank++;


***find following code

$query .= '`last_check_date` = NOW() ';
$query .= "WHERE `id` = {$link['id']}";
$gDb->query($query);
}
}
?>

***replace it with

$query .= '`last_check_date` = NOW() ';
$query .= "WHERE `id` = {$link['id']}";
$gDb->query($query);
$count_all++;
}

$query = "SELECT `value` FROM `{$gDb->mPrefix}config` WHERE `name`='site_email'";
$admin_email =& $gDb->getOne($query);

$date_check = date("F j, Y, g:i a");
$subject="Cron report [{$date_check}]";
$body = "{$count_all} links checked\n
(broken links checked - {$count_broken})\n
(reciprocal links checked - {$count_reciprocal})\n
(pagerank links checked - {$count_pagerank})";

mail($admin_email, $subject, $body, "From: {$admin_email}\r\n"."Reply-To: {$admin_email}\r\n");
}
?>


***Save and close it

Hope it helps.

[abtlang]

Thanks, John.

I'll try this and let you know (I imagine it will take a time so don't worry if you don't hear anything immediately).

Terry.

[John Turner]

You are welcome Terry.
Always at your services.

I'll be waiting for your results.

[Mark Brookes]

When developing the crons for ver 1.2 Vincent wrote some very good results logging procedures. I felt it was a shame that they were removed once the development phase was done.

In order to satisfy my host, and to not overload their server, I have had to tell check.php to only check 5 links at a time, so I could end up with 80+ emails telling me the results for each ckeck-run before the whole database of links had been checked.

What I would really like is something which said .... End of current major cycle, so send a email report of all results within that cycle- and highlight changes (eg new broken links, new non-reciprocal).

In fact, somewhere on the forum (or perhaps on the old forum) there is a discussion between Vincent & me of possible "follow-up actions" that the cron checks could/should do.

e.g if link is newly. broken, send link partner an email. If link has checked as broken twice, send link Partner an email saying that failure to correct may cause 'suspension' of link. If link has checked as broken 3 times send email & suspend link.
etc... etc.

(hint) I hope that ideas like these can be introduced in some future version :)


Regards
MArk

[abtlang]

Hi, John,

I've tested your solution for a few days, and I've seen that the only thing it really does is count the number of links each cron job checks.

What I'm really looking for is a way of knowing which are the broken links the cron job has found, and which faulty reciprocal links the job has found, that are no longer on the page where they should be.

Of the two, the broken links problem is not so bad, because there shouldn't be any broken links, so as soon as I see a link in the broken link list, then I know the cron job has found that one.

But the reciprocal links are a different problem. If some links are supposed to be reciprocal and others non-reciprocal, then if the cron job just changes the faulty ones it finds from "reciprocal links" to "non-reciprocal links", without notifying me which faulty link it found, then I'm not likely to spot it.

I can see two solutions:
1. A log sent to the directory administrator at the end of the cron job indicating which broken links were found and which faulty reciprocal links were found (not how many, but which ones).
2. If that is not possible, it might be feasible to create a new category of links, named "temp non-reciprocal links", and get the cron job to move any faulty reciprocal links it finds to there, rather than directly to "non-reciprocal links". (I seem to remember this solution from an earlier thread, but it seems to have disappeared - perhaps it's what Mark was talking about in his post).

Is there any chance of either of these two solutions being implemented?

Many thanks, as always,

Terry.

[John Turner]

Hi abtlang,

Sorry for delay.
In compliance with your requirements, here the solution:

It is necessary to hack your ../cron/check.php
***find following code

$query .= "OR (`last_check_date` IS NULL) ";
$query .= "ORDER BY `id` ASC LIMIT {$num_links}";
$links =& $gDb->getAll($query);

***add after it following code

$broken = '';
$nonreciprocal = '';


***find following code

if (empty($headers))
{
$link_header = 1;
}

***replace it with following code

if (empty($headers))
{
$link_header = 1;
$broken .= $link['url'].'\n';
}


***find following code

$content = get_page_content($url);
if (!empty($content))
{
$recip_valid = (int)has_url($content, $settings['reciprocal_text']);
}


***replace it with following code

$content = get_page_content($url);
if (!empty($content))
{
$recip_valid = (int)has_url($content, $settings['reciprocal_text']);
if($recip_valid == 0)
{
$nonreciprocal .= $link['url'].'\n';
}
}


*** find following code

$date_check = date("F j, Y, g:i a");
$subject="Cron report [{$date_check}]";
$body = "{$count_all} links checked\n
(broken links checked - {$count_broken})\n
(reciprocal links checked - {$count_reciprocal})\n
(pagerank links checked - {$count_pagerank})";


*** replace it with following code

$date_check = date("F j, Y, g:i a");
$subject="Cron report [{$date_check}]";
$body = "{$count_all} links checked\n
(broken links checked - {$count_broken})\n
(reciprocal links checked - {$count_reciprocal})\n
(pagerank links checked - {$count_pagerank})";
if($broken)
{
$body .= 'Broken \n'.$broken.'\n';
}
if($nonreciprocal)
{
$body .= 'Non reciprocal\n'.$nonreciprocal.'\n';
}


*** SAVE and close it.

You'll receive email with urls for broken and nonreciprocal links, if cron-verification will determine this.

Hope it helps.

[abtlang]

Thanks again, John.

As before, I'll try it out and let you know how it goes.

Best regards,

Terry.

[Mark Brookes]

Hello John,

An interesting development ... I have some questions :)

[1] My cron job is limited to 5 checks at a time to avoid overloading my host's server - It takes about 80 cron-jobs to complete checking ALL my links - is there any way to send me a summary email at the end of ALL the checks?

[2]
Please can you explain the numbers reported as

(broken links checked -)
(reciprocal links checked -)
(pagerank links checked -)

as they dont seen to tie up with the results which appear in the _links Table.

please see my attachments for a specific example

I think the numbes are about 'successful' results,
in which case I think
Broken Links Checked - Results OK = 4 (agreed)
Reciprocal Links Checked - Results OK = 3 (not 4 as reported in email)
Pagerank Links Checked - Results OK = 4 (agreed)

About the Warning reports:-

Warning these links are Broken- (ok)
* Please can it start each url on a new line

Warning these links fail Reciprocal Test:- (ok)
* Please can it start each url on a new line

*Please can there be report like....
Warning Partner homepage Pagerank < = 0



Regards
Mark

[abtlang]

Hello again, John.

First, thanks very much for the work done so far - now it is just what I wanted.

However, I have a slight problem - the email report doesn't include the broken links it finds. I notice in the attachment sent by Mark that his report does include a report on broken links found, so I assume I've made a mistake in my file. I attach a copy of my cron/check.php file after the changes I made - I wonder if you'd mind having a look at it to see if there's anything wrong.

And if you could answer Mark's request about how to get the broken links and reciprocal links reports onto separate lines, that would be the icing on the cake!

Thanks once more,

Terry.

[Mark Brookes]

Hello abtlang

I have compared your file & my file (see attached) and they are identical. This suggests to me that you should be receiving droken link details in your email.

* is it possible that because the report layout is currently a bit poor that the word "broken" is hidden among other text and you have missed seeing that some url's are actually 'broken' irls

* is it possible that none on your url's are broken?
Have you lookes at the ?_links table in phpMyAdmin, and does the 'header' field say 200 for every record/link - if so then every link is valid.


Regards
Mark

[John Turner]

Hi there,

There is an archived 'check.php' in the attachment.

It'll sent following information each cron job.


Subject:

Cron report [December 18, 2006, 4:08 pm]

Body:

10 links checked
=========================
2 broken links
-->Warning these links are Broken
http://link.url
http://link.url

3 non-reciprocal links
-->Warning these links fail Reciprocal Test
http://link.url
http://link.url
http://link.url

1 links PageRank not updated
-->Warning these links fail for Pagerank update
http://link.url



Hope this helps. :beer:

[Mark Brookes]

Hey John :beer:

Looking pretty good ... see attached

About the email report:
5 links checked
=========================
0 broken links
2 non-reciprocal links
-->Warning these links fail Reciprocal Test
http://www.sparklingspeeches.co.uk (http://www.sparklingspeeches.co.uk/)
http://www.freeweddingsites.com/

0 links PageRank not updated

About Reciprocals test:
*** in the above email, as non-reciprocal links = 2, then there should be 3 urls, listed as having failed the reciprocal test (?)

*** when I study the phpMyAdmin _links table I see 4 links with invalid reciprocals.(?)


About Pagerank test (see attachment2)
*** what does it mean when pagerank = -1 ?

*** Can the pagerank test work out if the PR = "GREY"?

As I understand it, if Google really dislikes a web site, it gives it a 'grey' PR, and anyone linking to it gets penalised also. In this case I would like to know if any of my links goto 'grey' urls.



Thanks for your perserverence

Regards
Mark

[abtlang]

Hi, John and Mark - and many thanks to the two of you for taking the time to help me out on this. :applause:

Thanks especially to you, John - the file looks excellent. As always, I'll probably take a couple of days to test it thoroughly enough to be able to make sure I've got everything set up right, but I can't imagine any problems this time.

I wish you both (and all eSyndicat users) a very Merry Christmas, and a New Year filled with peace, prosperity and good health,

Terry.

[John Turner]

You are welcome Terry.
Thank you for your congratulation.
I wish you Merry Christmas and Happy New Year too.

Mark,
*** in the above email, as non-reciprocal links = 2, then there should be 3 urls, listed as having failed the reciprocal test (?)

Cron informs you which links are falled checking only.

*** when I study the phpMyAdmin _links table I see 4 links with invalid reciprocals.(?)
I suppose that link were marked invalid before Cron checks it.
Try to change it's fields via phpMyAdmin.


*** what does it mean when pagerank = -1 ?

PageRank may have value from 0 to 10
If it's value equals -1, so there was a problem to update pagerank.

Hope it helps.

[Mark Brookes]

Hello John,

Cron informs you which links are falled checking only. I suppose that link were marked invalid before Cron checks it.
Try to change it's fields via phpMyAdmin.
[1]
Of Course silly me!
When I am looking at _links table values I am looking at the cumulative position, but the email rep[ort is about the results of "this time's tests"

To make this clearer to me & remind me I plan to change the mod to read:

5 links checked
=========================
0 New broken links
2 New non-reciprocal links
-->Warning these links fail Reciprocal Test
[...etc..]
0 links PageRank failed to update this time




[2]
PageRank may have value from 0 to 10
If it's value equals -1, so there was a problem to update pagerank. So in a situation where the url has a 'grey' PR what does check.php report?

As I understand it, for webmasters a link to a 'grey' PR page is much more significant (ie bad) than a link to a zero PR site, because it can actively penalise their own site's PR & SE ranking (?)

.
.
.

[Mark Brookes]

Hello again John,

I have fiddled with the email report layout along the lines I described.

I now get a nice layout like attachments 1 & 2


To help people who may not want to work through all the steps in the 2 modifications I attach the modified code which IS working for me :)


Thanks for all your work in getting this mod working.


Any possibility of a way to do a summary report of all the results in the current Major cycle - i.e after all the links have been checked?


regards
MArk

[John Turner]

Hi there,

There are two archives in the attachment

check.zip ====

Upload check.php to '../cron/' directory

It'll send email each cron job in this form

Subject:

Cron report [December 18, 2006, 4:08 pm]

Body:

10 links checked
=========================
2 broken links
-->Warning these links are Broken
http://link.url
http://link.url

3 non-reciprocal links
-->Warning these links fail Reciprocal Test
http://link.url
http://link.url
http://link.url

1 links PageRank not updated
-->Warning these links fail for Pagerank update
http://link.url


check_summary.zip ====

Contain:
check.php
update_cron.sql

Upload check.php to '../cron/' directory
Upload update_cron.sql to '../updates'
execute it via Admin Panel » Manage Database » Import

It'll send email when all links of your directory will checked in this form

Subject:

Cron cycle finished [December 18, 2006, 4:08 pm]

Body:

100 links checked
=========================
2 broken links
3 non-reciprocal links
1 links PageRank not updated



That's all.

[khunjack]

Thank you all !!!

That is fanatstic that you attached the mod file GREAT!!

[Mark Brookes]

Hello John,

Thank you for your work on a major-cycle summary.

Results so far for the new check_summary.zip files ...

Parse error: syntax error, unexpected T_LNUMBER in /home/finespee/public_html/wedding-resources-TEST/cron/check.php on line 243
which is $query .= '`cron_cycle`='1' ';

I note that the new cron_cycle field is all set to zero - "0" to start with.
Is that anything to do with the problem?

Regards
MArk

[John Turner]

Hi Mark,

This is my fault.

Please open check.php from 'cron_summary.zip'

***find following code

$query .= '`cron_cycle`='1' ';

***replace it with following code

$query .= "`cron_cycle`='1' ";


Thats all.

I have attached repaired version of cron_summary.zip

[Mark Brookes]

Hello John

This is my fault. No problem, testing is a necessary step :)


The new version results in

Error: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'BY `id` ASC LIMIT 5' at line 1
SELECT `id`,`url`,`reciprocal` FROM `FSlinksV20_links` WHERE ((`last_check_date` + INTERVAL 100 DAY < NOW()) OR (`last_check_date` IS NULL)) AND `cron_cycle`=0ORDER BY `id` ASC LIMIT 5



John,
I notice that the Summary-check.php seems to not email-report the urls that have failed the cron tests within the cycle. These items are the Administrator's Action Points.

Perhaps it is not practical to get a summary report similar to your new check.php(-with-email-report) ?
Maybe I should just use what you have kindly provided already ?

Is is worth your time working on an extra type of report that may only be of interest to me? after all your new check.php(-with-email-report) gives me the essential information (just 5 links at a time) :) ?
I would understand this.


Regards
Mark

[John Turner]

Ooops ... !

There is my fault.
So, I have attached 2 zipped archive.
cron_summary.zip (Updated)
cron_summary_ext.zip (Which send email after cycle is finished with full information about which links broken, non-reciprocal, pagerank not-updated)

Hope it'll be useful for all.

[Mark Brookes]

Hi John

Sorry to not report success ...

**The check.php in cron_summary.zip (Updated) will only run once and then stops.

So for example my settings mean it checks the first 5 links - it records the date in last_check_date; leaves cron_cycle values = 0. But when I try to run it from the browser again, nothing happens - no fields are changed & the program starts&stopr within 1 second. This suggests to me that it is checking something then deciding to quit processibng.


*** The check.php in cron_summary_ext.zip
If I substitute this check.php and try to run it the same happens.

In case I am doing something wrong/silly may I ask you to visit http://www.weddingspeeches.com/wedding-resources_TEST/ and see if you can see what is wrong?

Thanks
Mark

PS, the 'original' version of check.php does work properly so it is not a server problem this time

[John Turner]

Hi Mark,

I have fixed a few errors.
I have tested it on your TEST directory.
Thanks.
There are final version in the attachment.

[Mark Brookes]

Hello John,

Please explain.

Both cron_summary.zip & cron_summary_ext.zip have a file check.php in them.

How should I be using each of them?

Which one should I be using to get a summary email at the end of the cron_cycle.

Thanks
MArk

PS, Happy Christmas !! :)

[John Turner]

Hi there,

Like I wrote,

cron_summary.zip will be useful for get summary of each cron job.
cron_summary_ext.zip like cron_summary.zip but it also will send an email each cron cycle is finished.

Hope it helps.

[Mark Brookes]

Hello John.

Nice to see you back :) hope you have a happy Christmas....

Thanks for clarifying.
I have updated my TEST directory & tried out the files again: - The results are WONDERFUL!:good:


I have renamed the check.php files to:-
* check-MajorCycleReport.php
* check-MinorCycleReport.php
This way I can distinguish them from the basic check.php which comes with the download programs and which does not send any email reports.

Please may I ask for one more variant: -
The current check-MajorCycleReport.php sends both minor cycle reports and a summary major cycle report, This is probably great for people who can check say 100 links at a time, however to keep my host server happy I only test 5 links at a time and it takes 80 tests to complete the major cycle - SO, please could I have a variant which does not send minor cycle reports, but only the final major cycle report?



Finally:
May I suggest that these all be included in the next release of eSC. So that the administrators can choose what level of email reports they want, when they set up the cron job in cpanel.

ie. there would be 4 possible link checkiung crons for the Administrator to choose from:-
* check-noEmailReports.php
* check-MajorCycleReport.php
* check-MinorCycleReport.php
and hopefully :)
* check-MinorAndMajorCycleReport.php

Regards
Mark

[John Turner]

Hi Mark,

Here attached CRON archive, which reports only when whole cron cycle is finished.

In our V2.1 PRO this cron modification isn't included.

I am glad to help you.

[Mark Brookes]

Hi John,

Excellent,
Super,
Wonderful,
Great.
Works like a charm :good::friends:



... a final question to tidy up

does check.php check every link including disapproved, suspended, banned etc?
or does it just check Active links?

[John Turner]

Hi Mark,

Thank you for such kind words!
Always at your services.

does check.php check every link including disapproved, suspended, banned etc?
or does it just check Active links?
Cron checks all links.

[khunjack]

In our V2.1 PRO this cron modification isn't included.
I guess 2.1 will be the next version (current 2.0.01)

If that is the case...that concerns me!!!

[John Turner]

Hi khunjack,

You 'll be able to use cron report results modifications with V2.1
which is discussed in this thread.

[khunjack]

very good. Thank you John

[John Turner]

You are welcome khunjack,

Please feel free to contact me if you'll have question.

[Mark Brookes]

Hello John

Here is an idea for a refinement ....

can the check.php files please write a note to the Admin comment field for each link, like:-

2007-01-17 Cron Check reports: Broken=ok/NOT-ok; Reciprocal=ok/NOT-ok; HomePagePR=0123etc

regards
Mark

[John Turner]

Hi there,

CRON functionality will be increased in our V2.1 PRO
New options:
1. added CRON email reports
2. updated sponsored CRON
3. added backup CRON

:cool-yo:

[Mark Brookes]

Hi there,

CRON functionality will be increased in our V2.1 PRO
New options:
1. added CRON email reports <====
2. updated sponsored CRON
3. added backup CRON <====

Wow, what brilliant developments .... :D many congratulations to the clever, creative, wise, perceptive, and HANDSOME genius who thought them up

..... oh .... yes, of course and congratulations to the hard working genius who turned the ideas into something useful :)


Regards
Mark

[John Turner]

Thanks Mark for such kind words.
We are always at your services.