How to Make Your LLMS.txt Accessible to LLMS (OpenAI, Perplexity, etc)

Strategies for adding your llms.txt file to your website so that it gets indexed quickly (includes WordPress integration options). We are advocating for the option to provide the file directly to OpenAI or Perplexity or to request a crawl in the same way that you might do with Google Webmaster Tools.

How to Add LLMS.txt to a WordPress Website

Notify Crawlers and Bots

Include the file in robots.txt and add a directive in your robots.txt file to point to your llms.txt. For example:

<code>
User-agent: *
Allow: /llms.txt
</code>

Integrate with Your Sitemap Manually

Locate your sitemap URL (e.g., https://example.com/sitemap_index.xml) and use a text editor or plugin to include the llms.txt file in your sitemap:

<code>
<url>
<loc>https://example.com/llms.txt</loc>
<lastmod>2024-12-04</lastmod>
</url>
</code>

Save the changes and re-submit the sitemap to search engines and LLM platforms (e.g., via Google Search Console or directly to LLM platforms if supported).

Automate via Plugins

Use a hook in WordPress to dynamically append the llms.txt file to your sitemap if supported by Rank Math or Yoast SEO.

Leverage Internal Linking

• Link from High-Traffic Pages: Add a footer or header link to your llms.txt file to increase its visibility to crawlers.
• Use Schema Markup: Add metadata to signal the relevance of llms.txt:

  <code>
  <link rel="alternate" type="text/plain" href="/llms.txt" />
  </code>

Encourage External Crawling

• Publicize the File:
  • Announce the addition of your llms.txt file on social media or in industry communities.
  • Share the file’s purpose and availability in relevant forums and discussions, particularly for LLM users.
• Link from External Websites:
  • Collaborate with partners.
  • Add links in guest blogs or forum posts.

Optimize Crawling Speed

• Monitor Server Configuration: Ensure your server handles requests efficiently to avoid downtime or slow indexing.
• Fetch as Search Engine or LLM Platform: Use tools like Google’s “URL Inspection” to request indexing of specific URLs, including llms.txt.

Monitor and Iterate

• Use Tools:
  • Access Logs: Track when LLM crawlers access llms.txt.
  • Web Analytics: Monitor referral traffic from LLM-powered tools to confirm increased usage.

Implementation Example for WordPress

• For Rank Math:
  • Access the Rank Math Dashboard.
  • Navigate to General Settings > Sitemap Settings.
  • Use the “Additional Links” section to add your llms.txt URL.
• For Yoast SEO:
  • Access the Yoast SEO Settings.
  • Go to Search Appearance > General.
  • Under “Advanced settings,” manually add the llms.txt file if allowed.
• Add via Child Theme or Custom Plugin: Use the following PHP snippet in your WordPress child theme’s functions.php file to dynamically include llms.txt in your sitemap:

  <code>
          add_filter('wpseo_sitemap_entry', function($url) {
              $url[] = [
                  'loc' => home_url('/llms.txt'),
                  'lastmod' => date('c'),
              ];
              return $url;
          });
          </code>