The Crawl plugin is added...

[Sergey Ten]

Hello all,

I have just made the "crawl" plugin. Now you could download it. The plugin allow to grab the meta description, meta keywords and title of web site by URL and after add a listing to your directory.

[safatweb]

can i know the location as i could not find...

[Simon Gooffin]

safatweb
it will be available in few minutes in your downloads area

[royalmail]

Hi Sergey,

Is this an alternative or a better replacement for the spider plugin?

[smartpc]

grrrrr-8 esyndicat 2.2 gets better and better....

oh I see it works on backend, any chance can get this plugin to work on the suggest listing page....

[smicovic]

great job Sergey

[Sergey Ten]

Hi Sergey,

Is this an alternative or a better replacement for the spider plugin?

I think it is alternative plugin :) but you could compare both plugins yourself :)


grrrrr-8 esyndicat 2.2 gets better and better....

oh I see it works on backend, any chance can get this plugin to work on the suggest listing page....

David, do you mean in the admin panel? ok in any case I will make it asap.


great job Sergey

Thank you :)

[harivney]

Plugin is not what I expected as described in this post and it doesnt work at all. http://www.esyndicat.com/forum/showthread.php?t=12014&highlight=crawl

Note the request was for crawling multiple sites.(see post = url(s)

When I enter url in crawl plugin, I get error message = invalid url. Tried with http:// and without http: - same error

If crawl was designed to work with only 1 url than how is it any different than spider???

PS - Running 2.2 which I downloaded about a week ago. Doesnt seem to be anywhere in admin where I can tell which level of 2.2 I am running.

[Greg]

PS - Running 2.2 which I downloaded about a week ago. Doesnt seem to be anywhere in admin where I can tell which level of 2.2 I am running.
Look in the footer of any page in admin.

[tsands]

Plugin is not what I expected as described in this post and it doesnt work at all. http://www.esyndicat.com/forum/showthread.php?t=12014&highlight=crawl

Note the request was for crawling multiple sites.(see post = url(s)

When I enter url in crawl plugin, I get error message = invalid url. Tried with http:// and without http: - same error

If crawl was designed to work with only 1 url than how is it any different than spider???


This is exactly how I feel. I thought this plugin was going to grab multiple url(s), as well.

[safatweb]

Sergy pleas fix the category list, it confuses (if you have 600+ categories)

[Simon Gooffin]

I think we should make the same tree for categories like we have on suggest listing page...
safatweb, am I right?

[safatweb]

Yes same as "Submit Link" or "Move Link"

Thanks

[Sergey Ten]

Hello safatweb,

Ok. I will try to make it asap.

[safatweb]

Hello safatweb,

Ok. I will try to make it asap.

This is the main reason why i like esyndicat

[Simon Gooffin]

Sergey Ten
Any news on these changes Sergey?

[Sergey Ten]

Simon,

I will try to finish work today on this plugin.

[safatweb]

:) thanks Sergey

[Sergey Ten]

Hello all,

I remade the Crawl plugin. I added the tree categories menu and multi-parsing URLs functionality. Now you can download new version of plugin.

[pmadfm]

Hi,

Mervellous, great works !

It's possible to add the keep : meta name="reply-to" for email and meta name="location" for the country ? and blank if no present ...

Good Year !

[safatweb]

sergey :(, it takes a lot of time, & no result

can u check it again

[Sergey Ten]

Hello safatweb,

Hmm, so strange I have tested plugin on our test server and it worked ok. Could you send me your credentials via pm and I will check it.

[p0werm4n]

Hmm, I updated the crawl and now when I click on parse it just sits on "loading..." but it never loads anything. I am using firefox btw.

[Sergey Ten]

Hello p0werm4n,

Do you reupload all files? please check it so closely. I have changed all files in the plugin.

[pmadfm]

Hello Segey,

If you had time, it would be extra if you could add the meta “reply-to”, that would make it possible to recover the email for those which one fills.

Thank you very much.

[Sergey Ten]

Hello pmadfm,

I am sorry but I don't understand what do you mean :( What is "reply-to" meta tag?

[pmadfm]

Hello,

My English must be very bad (thank you Systran); -)

Example of META:
<title>01-AGF Annuaire G&eacute;n&eacute;raliste Francophone - liens durs rewrite</title>
<meta name="description" content="01-AGF, est un annuaire generaliste en lien durs et rewrite avec : description longue(500), Google Maps, Youtube, categories (20 000) : voisines, relatives, croisees, compte editeur, blog dedie, structure DMOZ ..." />
<meta name="keywords" content="01-AGF, annuaire, annuaire web, annuaire internet, site internet, liens durs, rewrite, site web, web, sites internet, generaliste, referencement, seo, francophone, lien dur, directory, sites internet, weblogs, blogs, fiche detaillée, ajoutez votre site, ajoutez un lien, referencez votre site, soumettre un site, repertoire, annuaire de site, classement" />
<meta name="author" content="&copy; SARL 01 ADFM - 2004 - 2008" />
<meta name="location" content="France, FRANCE" />
<meta name="reply-to" content="refer @ 01adfm.com" />

I would like only, if that is possible for you to add to the crawl the beacons META, Author, Location and reply-to.

By advance, thank you.

[juliusbarra]

Hi Sergey,

this plugin doesn't work well with french characters .

You already fixed this issue with spider plugin here


http://www.esyndicat.com/forum/post77735-35.html

and it works fine.

Is it possible to have the same mod in crawler plugin so it can works with french characters ?

Thanks
JuliusB

[Sergey Ten]

Hello JuliusB,

Try to do the following:

[ Open your admin/crawler.php ]
[ Find code ]


$content = esynUtil::getPageContent($url);


[ After add code ]


$content = mb_convert_encoding($content, "ISO-8859-1", "UTF-8");


[ Save change ]

[juliusbarra]

Hello Sergey,

I made the modification but, if before the characters like é è ç were changed to ? now they are deleted.

I changed your code to this
$content = mb_convert_encoding($content, "UTF-8");


and now it works perfect.

Can the change I made cause some issue?

Thanks
JuiusB

[Sergey Ten]

JuiusB,

I think no.