Search Engine Spider-Friendly JavaScript Content

If you have syndicated content delivered with JavaScript, the content might not be indexed by the search engines.

Syndicated content, in this article, means HTML content (text, images, forms, and so forth) delivered from a remote site and published on your page with JavaScript code similar to:

<script
   language="JavaScript"
   src="http://example.com/file.js">
</script>

If you syndicate the Possibilities articles, you're familiar with the above. The software used to prepare and deliver JavaScript syndicated content is Master Syndicator V4 from http://willmaster.com/msyn or Master Syndication Gateway V2 from http://willmaster.com/gateway

It make the syndicated content search engine spider-friendly, use an SSI tag to insert the content into your web page. The SSI tag runs a CGI program that fetches and delivers the content.

The CGI Program

Here is the CGI program that will fetch the syndicated content and send it to your web page:

#!/usr/bin/perl
my $JavaScriptURL = 'http://example.com/file.js';
use strict;
use LWP::Simple;
my $content = get $JavaScriptURL;
$content =~ s/^\s*<!--//s;
$content =~ s!(?://)?\s*-->\s*$!!s;
my @content = split /[\r\n]+/,$content;
for(@content)
{
        unless(/^document/) { $_ = ''; next; }
        $_ =~ s/^document\.writeln\(\'//;
        $_ =~ s/'\)\;$//;
        $_ =~ s/' ?\+ ?'//g;
        $_ =~ s/\\'/'/g;
        $_ =~ s/\\\\/\\/g;
}
$content = join "\n",@content;
print "Content-type: text/html\n\n$content";
# end of script

Edit and save the above program using a plain text word processor like NotePad or TextWrangler.

   1.

      The first line must point to the location of Perl on your server.
   2.

      On the second line, replace the URL between the apostrophes with the URL of the syndicated content. This would be the URL found in the src="_________" attribute of the syndication <script...> tag. (To syndicate WillMaster Possibilities articles, use the http://willmaster.com/possibilities/c/wmp.js URL.)

Upload the program to your server as ASCII/plain text, to a directory that can run CGI scripts. Give it 755 permissions.

It requires the Perl module LWP::Simple. If you are unsure whether or not your server has the module installed, ask your hosting company or use Master Pre-Installation Tester from http://willmaster.com/mpits

The SSI Tag

Here is the SSI tag that will run the CGI program:

<!--#exec cgi="/cgi-bin/programfile.cgi"-->

Replace "/cgi-bin/programfile.cgi" with the location and name of the CGI program on your server.

Use the SSI tag instead of the JavaScript syndication code.

On many servers, pages with SSI need to have file names with the .shtml extension. Your server may be different. Contact your hosting company if in doubt.

For additional information about SSI, see the "Server Side Includes" article linked from http://willmaster.com/possibilities/archives/

When you've installed the CGI program and use the SSI tag in lieu of the JavaScript syndication code, search engine spiders can index your syndicated content as if it was static content.