How Search Engine Crawlers (or Spiders) Work

Search Engine Crawlers, or Spiders, are small automated software programs that are sometimes also referred to as “robots”    These programs will frequently roam around the “world wide web” searching for new, updated, or changed web pages and helps search engines index or “catalog” every web site correctly to produce optimal search results for search engine users.  Some websites are “crawled” daily, while others are not.  

When a search engine spider arrives at your web page, it first looks for a robots.txt file. This is an transparent HTML code file used to tell spiders/robots which areas of your site are off-limits and shouldn’t be cataloged. Some examples would be pages that contain HTML code that are a waste of time (such as Flash pages). A robots.txt file will re-direct spiders away from these types of pages.

Search Engine Crawlers also collects outbound links from the page. These routes will eventually be followed to other pages. Spiders follow links from one page to another page.  Frequency of visits will vary from one search engine to another, as they are their databases are different from one another.

As a web site owner, you should regularly check and see which pages of your site have been visited by the crawlers and spiders.  You can do this by checking your log statistics program.  Most spiders are easily identifiable by their names.  

Any web site owner should know which pages the search engine robots have visited. Look at your server log reports or the results from your log statistics program. ( If you don’t have one, upgrade your web hosting service.   VectorInter.Net provides these tools free, with every web site hosting contract.) Most spiders / robots are easily identifiable by their “user agent” names. Google’s robot is named “Googlebot”. Other spiders have funny names, such as “Slurp”, Inktomi robot’s name.   

For helpful ranking factors and tips on how to make your website more visible to crawlers, check out


Learn the Basics of How Pay-Per-Click (PPC) Marketing Works

Are you having a
difficult time with your SEO strategy and struggling to boost your search
engine ranking?  

Assuming you have a
budget of some sort to work with, PPC may be a viable option for your
business.    PPC (which
stands for pay-per-click),
is a model of internet marketing which advertisers pay a fee each time one of
their ads is clicked.

PPC at allows
advertisers to bid for ad placement in a search engine’s sponsored listings
when someone searches on a keyword that is related to their business or
service.  For example, you own a massage
therapy clinic in Toronto.  You place a
PPC campaign for the keyword “Registered Massage Therapy in Toronto”.   Every time someone searches that keyword
phrase in a search engine, your ad will appear as a sponsored listing.  When that ad is clicked, it sends the visitor
to your website and you pay a small
fee to the search engine.

Fees can quickly add up.  If the cost is $3 per click and 300 people
click on your sponsored ad for that month, your fees will total $900.  However on the other side of the coin, if 50
of those people who clicked your ad leads to a massage that costs $100, you
just made $5000 in sales –  a $4100
profit.  So your ROI can be quite good.

Building a successful
PPC campaign, from researching and selecting the right keywords, to organizing
those keywords into an organized campaign is very time consuming. If you find
yourself short on time, hiring a professional search engine marketer is
recommended as they are already well versed on how the whole process works, and
how to select the most relevant keywords without going broke.

Need professional advice from an SEO professional?  Contact The Best Media – Digital Marketing Solutions for all your marketing needs.

What Can You Do to Prevent Your Web Site From Being Scraped?

Content scrapers are websites that steal your content for their own websites or blogs without your permission This is usually performed by using automated software.  This software can not only take the information from an RSS feed, but it can post the same information on someone else’s blog like it’s a new post.  Real-estate listing portals are one of the most vulnerable websites to web scraping.  Property data listings and real estate pictures are very valuable and relatively easy to steal using web scraping bots
If you have some experience in blogging, you’ve probably already fallen victim to web scraping bots.  It almost feels like you have been intruded on – as though someone broke into your home and started taking things without asking.   So what can you do to stop this from happening?  What preventative measures can you take?
As a simple first step, ask them to take it down and stop stealing your content.  If they reply and agree, bonus.  If they don’t agree or simply wont even reply to your emails, some further action will be necessary.
Secondly, if you find that your web site content has been taken or re-posted without your permission, you can file a DMCA-complaint with Google and they will investigate your inquiry.  
Third option, use a duplicate content detection service.  These services, such as Copyscape, will flag any duplicate content found.
And lastly, monitor your logs frequently.   New user-friendly tools have been introduced in the past few years to detect and block unwanted visitors or IP addresses from entering your site.   WordPress offers a tool called Akismet that does just that.  If you have a WordPress site, its also recommended to install a trusted security plugin and setup automatic backups of your database.
For information on duplicate content and it’s possible consequences, read

SEO Tools That Will Help You and Your Website Ranking

When it comes to Search Engine Optimization, we can use all the help we can get.  Particularly if you are doing most of the legwork yourself and don’t have a lot of money to invest to hire a professional SEO expert.

There’s a ton of SEO tools out there – some more complicated than others to use.  However, there are tons of free and effective tools available that doesn’t require a university degree to figure out how they work.

Here are just a few of the suggestions that will help you monitor, and tweek your website to be as “SEO friendly” as possible.

1. Broken Link Detector – This helpful tool by Ninja Internet Marketers will scan through your website and highlight errors such as broken internal or external links.  Broken links are not good for your rankings, so be sure you don’t have any on your website

2. – is a simple application from BinaryPlex.  It works in conjunction with your Twitter account and will display a timeline of the links you (or your friends) have tweeted.  This helps to measure the impact a particular twitter account has on clicks/rankings 

3. Google’s PageSpeed Insights –  It’s a well know fact that page speed directly affects your search engine rankings in Google.  Simply by entering a URL, this tool will test the loading time and performance of your website for both desktop and mobile, and identify opportunities to improve

4. Browseo – This tool will display your website to you the way search spiders see it.  This can be helpful to see the hierarchy you’ve given to particular elements on your website so you can make changes accordingly if needed.

5. SimilarWeb – Interested to see how you measure up to your competitors?  This clever tool compares traffic between two websites.  

6. – This keyword tool will provide you new keyword alternatives and suggestions based on a single keyword that you provide.

7. Google Keyword Planner – By entering in keywords, this tool will return helpful stats to guide your keyword strategy including monthly search volume, competition, and even suggests terms you may not have considered.

8. QuickSprout Website Analyzer – This will give you a complete analysis on just about everything including SEO optimization, social media, internal/external links, speed, tags, keywords, and competitor comparisons.

Looking for a Digital Marketing Agency to help you with your company website?  Contact the professionals at The Best Media for proven results.

How Voice Search Software Will Change Our Lives

Although voice search technology has been around for over a decade now, its popularity and use has dramatically risen thanks to virtual assistant products like Alexa, Siri and Cortana that use machine learning algorithms and allows them to get better at what they do as time goes.   We have grown to use them for real day-to-day tasks. What was once a basic search tool, can now set your alarm clock, take notes, and learn from past responses and behaviour.  This is just the beginning.

Over the next few years, we will see virtual assistants that can buy concert tickets, book reservations at your favorite restaurant, recommend a good movie according to your likes, maybe even find an empty parking spot at the mall.   Verbal communication predates any written language out there.  So it’s not surprising that voice search feels more natural than typed search.   

According to MindMeld, in 2015 voice search jumped from zero to 50 billion searches a month.   With more than half of all voice queries coming from smartphones, the mobile boom has a lot to do with the rise in popularity.  With tiny screens and even tinier buttons, it’s sometimes difficult to navigate or even type – particularly if you have large thumbs.  Voice search presents an easier solution to getting the same information in an easier fashion.   

This opens up a new gate in respect to marketing marketing.  For example, just as we would bid / pay Google AdWords to be that top listing for “paint supplies” in text search results, with voice search, we may be competing for that one spot and become the virtual assistants’ preferred service. 

Voice search technology will undoubtedly continue to grow and eventually be a part of us; alerting us; and resolving queries and issues before they even happen.  It will be like having your very own “Jarvis” (Iron Man’s assistant) at your service 24/7.    Think of how much it can do for people with health issues or physical disabilities.  Let’s say by monitoring your vital stats remotely, a virtual assistant will be able to tell you that your blood sugar level is getting low, or that you’re beginning to have a stroke and automatically calls an ambulance.   The possibilities are endless.

For another great article on the latest technologies, visit

What To Do If Your Web Site Has Been Blacklisted.

Most web site owners only start to think about security after their site gets hacked or infected.  Getting blacklisted due to a security threat can be disastrous to your website or business.   Aside from the humility and loss of trust from your visitors, in addition to that, your web traffic will plummet and your search engine rankings will be on hiatus.

Quite often, website owners are unaware that their site has even been hacked – and by the time they find out, it’s too late.  This “malicious content” that was placed on your site will flag search engines and blacklist your web site because it sees you as a threat to your visitors.  Google alone blacklists over 10,000 websites every day.   Bing, McAfee, and Norton also have their own blacklist protocols.

If Google detects malware on your site and you get blacklisted as a result, visitors to your site will see a warning saying:   Warning: Visiting this site may harm your computer!

The website at contains elements from the site, which appears to host malware – software that can hurt your computer or otherwise operate without your consent. Just visiting a site that contains malware can infect your computer.

The first thing you will want to do is verify that your site has indeed been blacklisted.  You can do a lookup on any of these websites:

You will then want to scan your site for malware using your anti-virus software, or other malware detecting program.  Once you’ve identified the malware, remove it.  If you are using WordPress or a similar content management system, update it to the latest version immediately.  It’s advisable to remove all plugins and themes and reinstall them.

Next, change all your admin passwords.

You can request Google to review your site – but only after you have resolved the security issues.  Once they’re resolved, it’s time to redeem yourself!

Log in to the Google Search Console (Formerly Webmaster Tools).  Add your site on Google Search Console and follow all the instructions to verify your site.   Then click on “Request Review” in the Security section to get it re-verified by Google.  This process is not immediate and may take up to a week before you are clear and no longer blacklisted.

For more information on malware, malicious bots, and how to avoid them, read

E-mail Marketing Tips For Small Businesses

E-mail marketing is alive and well.  With the explosion of social media sites
today, some feel that e-mail a dying marketing tool.


E-mail still remains one of the most lucrative
marketing channels for businesses. It’s a free source of advertising and it’s
targeting an audience that has already shown interest in your brand by
subscribing to your email list or newsletter.   
Customers are more inclined to buy from you because a level of trust has
already been established.  E-mails and newsletters can help boost sales, given
that it’s done properly.
  You can
actually lose customers if not done with care.

Here are some helpful tips to remember before sending out that email to
your subscribers.



1. Personalization
–Avoid generic greetings such as “Hello”, or “Dear Valued Customer”.  E-mails that address readers by their first
name are much more effective and helps build the relationship on a more
personal level.  Most email marketing
tools will have a function that will automatically replace each email with the
individuals’ name, so you don’t have to type all the names in manually every


2. Know your
Message, and Write It Well
–  Before
you start writing that email, ask yourself why you’re emailing your customers
and have a call to action (What you want them to do?)   Be sure it’s well written and is engaging
for your customers to read.  Assume that
you have a couple of seconds to grab their attention and make the first few
sentences enticing so customers will continue reading on.  Use headings and small paragraphs for ease of


3. Frequency
and Timing
–  Don’t over-do it.  It’s recommended that small businesses should
be emailing their subscribers once or twice a month.  Emailing too frequently will ultimately lead
to higher unsubscribe rates. Social media sites are better tools for frequent
communications.  Also keep in mind that
there are “prime” times to send these emails. 
Although it will vary slightly depending on your target audience and the
type of business you are in, typically these are the best time to marketing


Day of the week – Monday to Friday (except for mail about
recreational content, when it’s the weekend)

Time of Day – the optimal send time is 10.30am (except for
content about hobbies, which is 8.00am)


4. Do Not Spam – It’s very important that
you do NOT add people to your email list who did not personally sign up.
Remember, emails are great tools for maintaining and strengthening
relationships (or sales), but highly ineffective at recruiting new customers.
Adding unsuspecting names to your email list will only foster a negative
impression and might even get you dropped by your email provider. So don’t do
it!    Avoid
using spammy sounding phrases in the subject line.
  Include the business address, as this will
also help your mail from being treated as spam by email filters and servers.

5. Make it Mobile-friendly – Your emails
should be mobile friendly Prior to finalizing your next email campaign, make
sure that it is tested against all major mobile devices. Mobile emulators are
also available so you can see how the email looks on different devices. 

6. Use Social Media – Your website is not
the only medium to increase your email subscribers. Make full use of your
social media business accounts.  Include
your social media profiles in the email. Provide links to your Twitter feed, Facebook
page, and/or your blog in all email messages.  
Ask your followers to register and explain the benefits of doing so. 

7. Keep It Interesting – People get bored
easily, so keep your content static. 
Don’t focus on the same message over and over. Mix up your content and
keep your subscribers engaged by offering special offers and vouchers.  On average, 7 in 10 people make use of a
coupon or discount from a marketing email.

Check out these handy tips before creating your marketing emails and articles: