Marketo blocks image for Twitter card | Community
Skip to main content
Crystal_Pacheco
Level 3
June 19, 2019
Question

Marketo blocks image for Twitter card

  • June 19, 2019
  • 3 replies
  • 4722 views

Hi,
I created meta og tags for various social media platforms. The code itself is correct, but I get an error from Twitter saying it can't load the image because the robots.txt will not allow the Twitterbot to crawl. 

The image size for twitter is 300px x157px  and 129kb in weight.

<meta property="og:type" content="${og-type}">
<meta property="og:title" content="${og-title}">
<meta property="og:url" content="${og-url}">
<meta property="og:description" content="${og-description}">
<meta property="og:image" content="${og-image}">
<meta property="article:published_time" content="2019-04-10" />
<meta property="article:author" content="${og-author}" />

<meta name="twitter:card" content="summary_large_image">
<meta name="twitter:site" content="@Company">
<meta name="twitter:creator" content="@Company">
<meta name="twitter:title" content="${og-title}">
<meta name="twitter:description" content="${og-description}">
<meta name="twitter:image" content="${og-image-twitter}">


Error from Twitter is:
WARN: The image URL marketoURL.com/xx/xxx-xxx-xxx/images/twitter-image.jpg specified by the 'twitter:image' metatag may be restricted by the site's robots.txt file, which will prevent Twitter from fetching it.

I think Marketo is blocking the Twitterbot from crawling the image.

I know a solution would be to host the image externally.

Is there any way of having the image be hosted in Marketo and allowing twitter to crawl it? 

The robots.txt would look like this:
User-agent: Twitterbot
Disallow:

I'm not sure if I feel comfortable allowing the twitter bot to crawl all of Marketo. 

Any ideas?

3 replies

SanfordWhiteman
Level 10
June 19, 2019

You can customize the robots.txt for your LP domain.

Casey_Grimes2
Level 10
June 19, 2019

Hi Crystal,

By default, Marketo itself does not block images from being accessed by Twitterbot, so this is most likely being done through your overall corporate website's robots.txt vs. the Marketo subdomain. 

Crystal_Pacheco
Level 3
June 19, 2019

Is there any danger in putting this into the robot.txt file and uploading it to marketo and redirecting the marketo robots.txt file?

The robots.txt would look like this:
User-agent: Twitterbot
Disallow:

Crystal_Pacheco
Level 3
June 19, 2019

The current robots.txt file is:

User-agent: * Disallow: /rs/

Is that the default setting?

SanfordWhiteman
Level 10
June 19, 2019

That looks customized -- or maybe just different from the instances I've checked based in instance age. Feel free to replace with what's right for your situation!