Booking bugs cluster in the gap between "the rule" and "the moment the rule is checked". Closing that gap is most of the work. The calendar widget is the easy part.

Define each service before you touch pricing

Write one row per bookable thing and fill in every field, even the ones that feel obvious:

• Duration in minutes as it is delivered in practice, not as advertised.
• Buffer after (cleanup, notes, travel) and buffer before, if any.
• Capacity: how many bookings may overlap — one chair, one room, three staff, twelve seats.
• Resource: the person, room, or equipment the booking consumes.
• Price, and whether a deposit is required to hold the slot.
• Cancellation window, and what happens to the deposit inside it.
• Location: address, meeting link, or "we come to you" with a service radius.
• What the customer must bring or prepare.

The buffer field is the one people skip and regret. A 60-minute treatment with a 15-minute turnaround is a 75-minute block on the calendar. If the system only knows about the 60, every back-to-back booking runs late by mid-afternoon and the last customer of the day pays for it.

The time rules that decide whether the calendar is honest

Store the business timezone once, at the account level. Render times in the visitor's timezone if you like, but every confirmation, reminder, and admin screen should print the business timezone next to the time.

Then the rules themselves:

• Opening hours per weekday, plus exceptions: holidays, one-off closures, a short day before a holiday.
• Slot granularity. Fifteen-minute steps with a 50-minute service produce a grid where a 10:00 booking blocks 10:15, 10:30, and 10:45 as start times. Decide whether that is what you want, because customers will read it as "the 10:15 is taken".
• Minimum notice. Can someone book in ten minutes, in two hours, next day?
• Booking horizon. How far ahead the calendar opens — 30 days, 90, a year.
• Per-resource availability. Two staff working Saturdays and one working Mondays is a rule, not a note in someone's head.
• Same-day cutoff for changes, which is often different from the booking cutoff.
• Partial-day closures, like a two-hour staff meeting that should not close the whole day.

Two edge cases are worth deciding explicitly, because both generate support tickets.

The last slot of the day. A booking that ends exactly at closing time is valid. Systems that compute "start + duration < closing" reject it, and the customer sees a calendar that stops an hour early for no visible reason.

Daylight saving. On the spring-forward day, 02:30 does not exist; on the fall-back day it exists twice. If times are stored as local wall-clock strings, one of those bookings is wrong and the reminder fires at the wrong hour. Store instants in UTC with the timezone as a separate field and the problem disappears.

Availability has to be computed when it is asked for

A slot list generated at midnight and cached for the day is wrong by 09:05. Compute availability per request from the rules, the existing bookings, and the current time.

The other half is holding the slot while the customer is on the payment page. A hold is a row with a status and an expiry:

status:     pending | confirmed | cancelled | completed | no_show
starts_at:  timestamptz
expires_at: timestamptz, only set while pending

pending and confirmed both consume the slot. pending expires after ten or fifteen minutes and a scheduled job clears it. Every other status releases it.

Then let the database refuse the double booking instead of trusting the check you just ran in application code:

create unique index bookings_slot_unique
  on bookings (resource_id, starts_at)
  where status in ('pending', 'confirmed');

That is PostgreSQL syntax for a partial unique index. MySQL has no where clause on indexes, so there you either include the status in the key via a generated column or wrap the insert in a transaction that re-checks the slot before committing. Either way the second concurrent request gets an error you can turn into "that slot just went, here are the next three".

The confirmation email is the product

Most complaints about a booking system are complaints about the confirmation email. It needs:

• Date, start time, end time, and the timezone label.
• Service name and duration, price with tax, and the deposit shown separately from the balance.
• The address with a map link, or the meeting link — not both as an afterthought.
• The cancellation deadline as a date and time, not as "24 hours' notice", plus the link that actually performs the cancellation.
• A phone number or reply address that a human answers.
• An attached calendar file.

Build the attachment properly: a stable UID per booking, DTSTAMP in UTC, DTSTART and DTEND in UTC, METHOD:REQUEST, and a SEQUENCE you increment. If the customer moves the booking, resend with the same UID and a higher SEQUENCE. If it is cancelled, send METHOD:CANCEL. A fresh invite with a new UID leaves the old event sitting in the customer's calendar next to the new one, and they show up on the wrong day.

Send a reminder 24 hours out and a second one a few hours before. Both should carry the cancel link. A reminder with no exit is a reminder that produces a no-show.

Check the email as plain text, on a phone, in a client that blocks remote images.

What to test before it goes live

Work through these as scenarios, not as pages:

• Two browsers, same last slot, submitted at the same moment. One gets a confirmation, the other gets a clear message. No 500 and no silent overwrite.
• The last slot of the day, booked at closing time minus the service duration. It should be accepted.
• A booking made by someone in a different timezone. The confirmation shows the correct local time for the business.
• Cancel exactly at the deadline, and one minute past it. Behaviour matches the policy in both directions, and the deposit path matches too.
• Payment abandoned. The hold expires, the slot returns to the calendar, no confirmation goes out.
• Double-tap on the submit button. One booking, one email.
• A holiday and a half-day closure. Neither is bookable, and the day still renders without gaps that look like bugs.
• The email in a spam check, and the .ics opened in two different calendar applications.
• An admin moving a booking. The customer receives an update and the old event is replaced, not duplicated.
• A refund or a no-show. The slot is released and the record still shows what happened and who decided it.

Write the policy answers down — cancellation deadline, deposit refundability, lateness, no-show fee — and put the same text in the email and on the booking page. When the two disagree, the customer will quote the email back at you, and they will be right.

llms.txt is a plain markdown file at the site root: a title, a one-line description of the site, then grouped links with a short note on each. It was proposed in 2024 as a way for language models to read a curated map of a site instead of inferring one from HTML. No major assistant has committed to fetching it on a schedule, and there is no way to confirm that an answer engine used yours.

So the file is a cheap bet, not a switch. What follows is how to make the bet a good one on WordPress: the ordering, the generation, the caching, and the checks that tell you whether anything is happening at all.

What it does not replace

robots.txt says who may crawl. sitemap.xml lists what exists. llms.txt says what matters and why. They sit next to each other and do not conflict — but they do not cover for each other either. If your robots.txt disallows a crawler, a perfect llms.txt will not get that crawler in the door. If a page is noindex, leave it out of llms.txt too; listing it only advertises a page you have decided not to surface.

Put the pages in the order a buyer would ask for them

The common mistake is generating the file from a query in date order. That produces a reverse-chronological blog roll, which is the least useful ordering possible for someone trying to work out what you sell.

Order by the questions, not by the CMS:

• What this company is and what it does — homepage, and an about page only if it is short and specific.
• What it costs and how to start — pricing, how it works, the contact or quote form.
• The individual services or products, one line each.
• Reference material that answers objections — FAQ, docs, a page that explains the difference between two things you offer.

Leave out: category and tag archives, paginated duplicates, internal search results, staging, gated PDFs, and three-paragraph news posts. A short file with strong entries beats a complete one.

Each entry is one line, and the sentence after the colon is the part that earns the read:

- [Setup service](https://example.com/setup): We install and configure the template on your host, migrate your content, and hand over with a short editing guide.

Write that sentence for someone who has never seen the page. Do not paste the meta description if the meta description is keyword soup.

One URL per line, once. If a page belongs in two groups, keep it in the earlier group — repetition costs tokens and reads as padding.

Serving it from WordPress when the docroot is read-only

Managed hosts often give you no writable web root, and if a CDN sits in front, a file you upload can be cached longer than you expect. The dependable path on WordPress is a rewrite rule that renders the file per request.

Drop this into a mu-plugin and activate nothing — files in wp-content/mu-plugins/ load automatically:

<?php
// wp-content/mu-plugins/llms-txt.php

add_action( 'init', function () {
    add_rewrite_rule( '^llms\.txt$', 'index.php?llms_txt=1', 'top' );
} );

add_filter( 'query_vars', function ( $vars ) {
    $vars[] = 'llms_txt';
    return $vars;
} );

/**
 * The editorial layer: page IDs, grouped, in the order they should appear.
 */
function my_llms_groups(): array {
    return [
        'Start here' => [ 12, 14 ],   // home, how it works
        'Pricing'    => [ 19 ],
        'Services'   => [ 31, 32, 33 ],
        'Answers'    => [ 44, 45 ],
    ];
}

function my_llms_summary( WP_Post $post ): string {
    $text = $post->post_excerpt !== '' ? $post->post_excerpt : $post->post_content;
    $text = wp_strip_all_tags( strip_shortcodes( $text ) );
    $text = trim( preg_replace( '/\s+/u', ' ', $text ) );

    if ( mb_strlen( $text ) > 160 ) {
        $text = mb_substr( $text, 0, 157 ) . '...';
    }

    return $text;
}

function my_llms_render(): string {
    $out  = '# ' . get_bloginfo( 'name' ) . "\n\n";
    $out .= '> ' . get_bloginfo( 'description' ) . "\n\n";

    foreach ( my_llms_groups() as $group => $ids ) {
        $lines = [];

        foreach ( $ids as $id ) {
            $post = get_post( $id );
            if ( ! $post || 'publish' !== $post->post_status ) {
                continue;
            }
            if ( '1' === get_post_meta( $id, '_my_llms_exclude', true ) ) {
                continue;
            }
            $lines[] = sprintf(
                '- [%s](%s): %s',
                wp_strip_all_tags( get_the_title( $post ) ),
                get_permalink( $post ),
                my_llms_summary( $post )
            );
        }

        if ( $lines ) {
            $out .= "## {$group}\n\n" . implode( "\n", $lines ) . "\n\n";
        }
    }

    return rtrim( $out ) . "\n";
}

add_action( 'template_redirect', function () {
    if ( '1' !== get_query_var( 'llms_txt' ) ) {
        return;
    }

    $cache_key = 'my_llms_txt';
    $body      = get_transient( $cache_key );

    if ( false === $body ) {
        $body = my_llms_render();
        set_transient( $cache_key, $body, HOUR_IN_SECONDS );
    }

    header( 'Content-Type: text/plain; charset=utf-8' );
    header( 'X-Robots-Tag: noindex' );
    echo $body;
    exit;
} );

add_action( 'save_post', function () {
    delete_transient( 'my_llms_txt' );
} );

After adding the file, open Settings → Permalinks and save once. The rewrite rule only lands after the rules are flushed; without that step /llms.txt keeps 404ing and template_redirect never sees the flag.

On nginx the request has to reach PHP at all:

location / {
    try_files $uri $uri/ /index.php?$args;
}

Verify from outside: curl -sI https://example.com/llms.txt | head -n 5 should show content-type: text/plain, and curl -s https://example.com/llms.txt | head -n 20 should show markdown, not HTML.

The transient exists because rendering walks post IDs and runs content filters on every one of them. An hour is fine, and save_post clears the cache so an edit appears on the next request. If a CDN caches .txt aggressively, purge that single path on save rather than waiting for the TTL.

Keeping it from drifting

A file rots in two ways. Hand-written, it drifts the moment someone changes a slug. Fully automatic — every published post, newest first — it becomes a changelog nobody wants.

The version above sits between those: the ID list is hand-maintained, the rendering is automatic. Maintenance is a quarterly task of ten minutes. Open the email you send most often to prospects and check that every link in it appears in the file, in roughly the same order. Anything you would not put in front of a prospect does not belong in the file.

llms-full.txt is usually a mistake

The companion idea — dump the entire site as markdown into llms-full.txt — is easy to generate and expensive to consume. A template-rendered page repeats header, nav, footer, and sidebar on every URL, so a 200-page site can produce a file where most of the bytes are identical chrome.

If you publish one anyway:

• Strip the theme chrome and render the content area only.
• Dedupe by URL, and drop pages that fail the same "would I show this to a prospect" test.
• Reference it from the main llms.txt as a single line at the bottom, nothing more.
• Generate it on demand rather than on a nightly cron that leaves a stale copy sitting around.

Measure the file with a tokenizer before you publish it and decide what you are willing to spend. A file nobody finishes reading has no advantage over a short one.

Proving whether anything fetches it

Everything above is unverifiable until you look at requests. Start with the file itself:

sudo zgrep -h 'llms\.txt' /var/log/nginx/access.log* \
  | awk '{print $1, $9, $12, $13, $14}' | sort | uniq -c | sort -rn | head

Then narrow to the agents you care about:

sudo grep -iE 'GPTBot|OAI-SearchBot|ClaudeBot|Claude-User|PerplexityBot|Google-Extended' \
  /var/log/nginx/access.log | tail -n 50

User agents are trivially spoofed, so treat the count as a hint rather than a measurement. What carries information is the shape: a few hits from one agent, repeating weekly, is more interesting than a thousand from a scraped list. If your CDN terminates TLS, the origin log may show nothing at all and you need to read the CDN's log instead.

A 200 does not mean the file was read, and a read does not mean it was used. What you can check is whether the entries are correct, whether anything requests the file, and whether the pages it points to are the pages you actually want quoted.

If you would rather not maintain the PHP, AumViso generates llms.txt alongside schema, sitemap, and meta output. Choosing and ordering the pages stays an editorial job either way.

The weight problem is rarely spread across forty photos. It's almost always one hero image and one autoplaying video, and the rest of the page is fine. Fix those two and leave the detail shots alone.

What the phone actually downloads before it shows anything

The first screen needs the HTML, the CSS, and one image — the largest element in view, usually the hero. That's the LCP element. A workable mobile budget is under roughly 500 KB for those three combined, with the hero visible in under 2.5 seconds on a mid-range Android over 4G. Everything below the first screen is a separate problem and should not be in that first request.

That's the whole mental model. A factory tour page with twelve photos, a poster frame and a click-to-play walkthrough can sit around 1.5–2.5 MB total and still paint fast, as long as the hero isn't a 4 MB camera JPEG.

Which photos earn a place on the page

A buyer is checking capability, not browsing a gallery. The set that does the work is small:

• the building from outside, with signage or the loading bay in frame
• one wide shot of the floor, taken from a corner so depth reads
• a close-up of the operation the buyer is paying for
• the QC bench with instruments in use, not arranged
• packing and labelling, because that's where damage claims come from
• the nameplate or serial label of the main machine
• raw material storage, if lead time depends on it

Each one gets a caption with a specific: line number, machine model, tonnage, the month it was taken. Captions are cheap and they're what separates a real page from a stock template.

Cut near-duplicates without mercy. Also cut anything with a customer's part number, a competitor's logo on a box, or a whiteboard with a production schedule on it. Those are the photos that create problems later, and they cost bytes as well.

Export at display size, not camera size

A 6000px phone photo displayed at 760px wide is sending five times the pixels the screen can use.

• hero: 1600–1920px wide, one size for desktop, one at 800px for phones
• inline photos: 1200px wide maximum
• thumbnails and grid tiles: 400px
• JPEG at quality 78–82, or WebP around 75, or AVIF if your encoder handles it — with a JPEG fallback

Strip the metadata, which also removes GPS coordinates from photos taken at the plant:

exiftool -all= -overwrite_original *.jpg

Keep the colour natural. If you want a consistent look, apply one mild grade to the set as a whole, not a per-image preset — uniform heavy grading is one of the fastest ways to make genuine photos read as stock.

Markup that doesn't shift the layout

Give the browser the dimensions and the candidate sizes, and it will pick one file and reserve the space before the image arrives.

<picture>
  <source type="image/avif"
          srcset="/img/floor-800.avif 800w, /img/floor-1600.avif 1600w"
          sizes="(max-width: 720px) 100vw, 760px">
  <source type="image/webp"
          srcset="/img/floor-800.webp 800w, /img/floor-1600.webp 1600w"
          sizes="(max-width: 720px) 100vw, 760px">
  <img src="/img/floor-1600.jpg"
       srcset="/img/floor-800.jpg 800w, /img/floor-1600.jpg 1600w"
       sizes="(max-width: 720px) 100vw, 760px"
       width="1600" height="900"
       alt="Three machining cells on line 2, seen from the aisle"
       fetchpriority="high" decoding="async">
</picture>

Preload that one image in the head, and only that one:

<link rel="preload" as="image" type="image/avif"
      imagesrcset="/img/floor-800.avif 800w, /img/floor-1600.avif 1600w"
      imagesizes="(max-width: 720px) 100vw, 760px"
      href="/img/floor-1600.avif">

Every image below the fold gets loading="lazy" decoding="async" and no preload. Never lazy-load the hero — deferring the LCP element is the single most common way a fast site still measures slow.

Video is where the page weight actually goes

A three-minute 1080p walkthrough is 150–400 MB off a camera. Even after "compression" it's often 20 MB, and it's sitting behind the headline.

Split it into two assets. A six-to-ten-second silent loop near the top, around 1.5–2.5 MB at 720p, and the full walkthrough as click-to-play further down the page or on its own page. Nobody watches three minutes standing in a trade fair aisle, but plenty of people click a play button when they've already decided they're interested.

# 8 second silent loop, 720p, progressive download
ffmpeg -ss 00:00:04 -t 8 -i walkthrough.mov \
  -vf "scale=1280:-2,fps=25" -an \
  -c:v libx264 -profile:v high -crf 26 -preset slow \
  -movflags +faststart line2-loop.mp4

# VP9 version for browsers that take it
ffmpeg -ss 00:00:04 -t 8 -i walkthrough.mov \
  -vf "scale=1280:-2,fps=25" -an \
  -c:v libvpx-vp9 -crf 34 -b:v 0 -row-mt 1 line2-loop.webm

# poster frame, taken from the video itself
ffmpeg -ss 00:00:05 -i walkthrough.mov -frames:v 1 \
  -vf "scale=1280:-2" line2-poster.jpg

Check the result with ls -lh line2-loop.*. If the mp4 is over 3 MB, raise the CRF before you shorten the clip.

<video class="loop" autoplay muted loop playsinline preload="none"
       poster="/img/line2-poster.jpg" width="1280" height="720">
  <source src="/video/line2-loop.webm" type="video/webm">
  <source src="/video/line2-loop.mp4" type="video/mp4">
</video>

preload="none" plus poster means the poster shows immediately and the file is fetched when the browser starts playing it. If you want nothing downloaded until the visitor asks, drop autoplay and put a play button over the poster. On small screens, or when the visitor has asked for reduced data, show the poster only:

@media (max-width: 600px) {
  video.loop { display: none; }
  .loop-poster { display: block; }
}

@media (prefers-reduced-motion: reduce) {
  video.loop { display: none; }
  .loop-poster { display: block; }
}

Two things not to do: don't use a GIF of the machine running — the same clip as H.264 is an order of magnitude smaller; and don't put a full-screen background video behind text on mobile, where it competes with the hero image for bandwidth and destroys legibility in daylight.

Serving, so the second visit is free

Fingerprint the filenames (floor-1600.9f2c1a.jpg) and cache them permanently:

Cache-Control: public, max-age=31536000, immutable

HTML stays no-cache. Text assets get Brotli or gzip; images and video are already compressed and shouldn't be re-encoded on the fly. If images are being served through a PHP script or a resize endpoint, move them to static files — the script alone can add hundreds of milliseconds before the first byte of the image.

A slider or carousel at the top of the page makes everything above harder: most implementations load every slide up front. A single static hero with a thumbnail grid underneath is easier to keep fast and easier for a buyer to scan.

How to check it on the connection your buyers have

Lighthouse in mobile mode with throttling, and read the LCP element and the transferred bytes rather than the score. Then do it once by hand: DevTools network throttling on Slow 4G with 4x CPU slowdown, reload, and watch when the hero appears and whether the text jumps as it does. The office Wi-Fi test tells you nothing, because your buyers are on a phone in a hall with two bars.

If the layout moves while loading, an image is missing width and height, or a font is swapping late. Both show up as CLS in the same report.

Keeping it real while keeping it light

Specifics are what make a page look genuine: a caption naming the line and the month, a person in frame, a forklift that's slightly in the way. Leave the imperfect wide shot in — it reads as a place rather than a set. Keep one wide and one close-up per process instead of fifteen angles of the same machine, and prefer a phone photo taken on the floor over a stock image of somebody else's factory, as long as it's sharp and correctly exposed.

A short checklist before it goes live

• hero exported at 1600–1920px, under 300 KB, AVIF or WebP with a JPEG fallback
• only the hero is preloaded and marked fetchpriority="high"; everything else is lazy
• every image has width and height
• EXIF stripped, including GPS
• the top video is a silent loop under about 2.5 MB with a poster frame from the clip
• the full walkthrough is click-to-play, not autoplaying
• no GIFs
• image filenames are hashed and cached for a year
• tested once on a real mid-range Android over mobile data

The fix is not a better webhook call. It is accepting that the submission is saved the moment it arrives, and that delivery to the CRM is a separate job that can fail, retry, and be replayed without the visitor seeing a thing.

Save first, send later

The submit handler has one job: validate, write a row, answer the visitor. No outbound HTTP inside the request someone is waiting on.

add_action('wp_ajax_nopriv_lead_submit', 'handle_lead_submit');
add_action('wp_ajax_lead_submit', 'handle_lead_submit');

function handle_lead_submit() {
    check_ajax_referer('lead_form');

    $name  = sanitize_text_field($_POST['name'] ?? '');
    $email = sanitize_email($_POST['email'] ?? '');
    $phone = sanitize_text_field($_POST['phone'] ?? '');
    $note  = sanitize_textarea_field($_POST['message'] ?? '');

    if ($name === '' || !is_email($email)) {
        wp_send_json_error(['message' => 'Name and a valid email are required'], 422);
    }

    global $wpdb;
    $table = $wpdb->prefix . 'lead_queue';

    $wpdb->insert($table, [
        'idem_key'        => wp_generate_uuid4(),
        'payload'         => wp_json_encode([
            'name'    => $name,
            'email'   => $email,
            'phone'   => $phone,
            'message' => $note,
            'source'  => 'contact-form',
            'sent_at' => gmdate('c'),
        ]),
        'status'          => 'pending',
        'attempts'        => 0,
        'next_attempt_at' => gmdate('Y-m-d H:i:s'),
    ], ['%s', '%s', '%s', '%d', '%s']);

    wp_send_json_success(['reference' => $wpdb->insert_id]);
}

The table it writes into:

CREATE TABLE wp_lead_queue (
  id BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
  idem_key CHAR(36) NOT NULL,
  payload LONGTEXT NOT NULL,
  status ENUM('pending','sent','dead') NOT NULL DEFAULT 'pending',
  attempts TINYINT UNSIGNED NOT NULL DEFAULT 0,
  last_error VARCHAR(500) NULL,
  next_attempt_at DATETIME NOT NULL,
  created_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
  sent_at DATETIME NULL,
  PRIMARY KEY (id),
  UNIQUE KEY idem_key (idem_key),
  KEY status_next (status, next_attempt_at)
);

The unique key on idem_key earns its place. A double-submit inserts twice only when the key differs, and every retry of the same row carries the same key to the CRM, which gives the receiving side something to deduplicate on.

What counts as failed

Three outcomes get logged as success by accident:

• The request threw. wp_remote_post returns a WP_Error on timeout or DNS failure, and plenty of handlers never look.
• The status code was 4xx or 5xx. Retrying a 400 with an identical body will keep failing; that row needs a human, not another attempt.
• The status was 200 and the body says {"ok": false}. APIs report field-level rejections this way all the time.

Write the rules down and code against them: 2xx plus a body that does not declare an error means delivered. 4xx other than 429 means dead, no retry. 429, 5xx and transport errors mean retry with backoff.

The worker

function aum_push_leads() {
    global $wpdb;
    $table = $wpdb->prefix . 'lead_queue';

    $rows = $wpdb->get_results(
        "SELECT * FROM {$table}
          WHERE status = 'pending'
            AND next_attempt_at <= UTC_TIMESTAMP()
          ORDER BY id ASC
          LIMIT 20"
    );

    foreach ($rows as $row) {
        $res = wp_remote_post('https://crm.example.com/api/leads', [
            'timeout' => 8,
            'headers' => [
                'Content-Type'    => 'application/json',
                'Idempotency-Key' => $row->idem_key,
            ],
            'body'    => $row->payload,
        ]);

        if (is_wp_error($res)) {
            aum_lead_retry($row, 'transport: ' . $res->get_error_message());
            continue;
        }

        $code = wp_remote_retrieve_response_code($res);
        $body = wp_remote_retrieve_body($res);

        if ($code >= 200 && $code < 300 && !aum_lead_body_is_error($body)) {
            $wpdb->update($table, [
                'status'  => 'sent',
                'sent_at' => gmdate('Y-m-d H:i:s'),
            ], ['id' => $row->id]);
            continue;
        }

        if ($code >= 400 && $code < 500 && $code !== 429) {
            $wpdb->update($table, [
                'status'     => 'dead',
                'attempts'   => $row->attempts + 1,
                'last_error' => substr($code . ' ' . $body, 0, 500),
            ], ['id' => $row->id]);
            continue;
        }

        aum_lead_retry($row, $code . ' ' . $body);
    }
}

function aum_lead_body_is_error($body) {
    $data = json_decode($body, true);
    return is_array($data) && array_key_exists('ok', $data) && $data['ok'] === false;
}

function aum_lead_retry($row, $error) {
    global $wpdb;
    $attempts = $row->attempts + 1;
    $delay    = min(3600, 30 * (2 ** $attempts)) + wp_rand(0, 30);

    $wpdb->update($wpdb->prefix . 'lead_queue', [
        'attempts'        => $attempts,
        'status'          => $attempts >= 8 ? 'dead' : 'pending',
        'last_error'      => substr($error, 0, 500),
        'next_attempt_at' => gmdate('Y-m-d H:i:s', time() + $delay),
    ], ['id' => $row->id]);
}

The jitter matters when a CRM has a bad minute. Twenty queued rows that all retry on the same second will all fail again on the same second.

A worker running every five minutes with a cap of twenty rows handles 200 leads an hour and gives a row eight attempts over roughly three hours before it lands in dead. If volume grows, raise the cap rather than shortening the interval.

add_filter('cron_schedules', function ($schedules) {
    $schedules['five_minutes'] = ['interval' => 300, 'display' => 'Every five minutes'];
    return $schedules;
});

add_action('init', function () {
    if (!wp_next_scheduled('aum_push_leads_hook')) {
        wp_schedule_event(time() + 300, 'five_minutes', 'aum_push_leads_hook');
    }
});
add_action('aum_push_leads_hook', 'aum_push_leads');

Traffic-driven cron skips runs on a quiet site, which is exactly when a lead from last night is sitting in the queue. If leads need to land within minutes, point system cron at wp cron event run --due-now every five minutes and keep both.

Duplicates

A retry after a timeout is the classic duplicate: the CRM saved the record, the response never came back, the queue tries again. Both defences are worth having.

Send an idempotency key on every attempt — same key for every retry of the same row. The CRM stores it and returns the original record instead of creating a second. Not every CRM honours the header, so keep a fallback: hash the normalised email plus phone and check for an existing record before insert on the receiving side.

Write the CRM's record id back into the row when it comes back. When someone asks whether a lead arrived, the answer becomes a lookup instead of a search through two systems.

Replaying the dead queue

Dead rows still hold a payload and a decision. Read the errors before touching anything:

wp db query "SELECT id, attempts, last_error, created_at FROM wp_lead_queue WHERE status='dead' ORDER BY id DESC LIMIT 50"

If the cause was a mapping change or an expired token, fix that first, then requeue:

wp db query "UPDATE wp_lead_queue SET status='pending', attempts=0, next_attempt_at=UTC_TIMESTAMP() WHERE status='dead'"

Anything still failing after the second pass is a conversation with whoever owns the CRM, not another retry loop.

Alert on the dead count. A daily email with the number and the top three errors is enough. A queue nobody watches is decoration.

Inbound webhooks deserve the same care

If the CRM calls back — status change, assignment, a note added — verify the signature against the raw body before parsing:

$raw  = file_get_contents('php://input');
$sig  = $_SERVER['HTTP_X_SIGNATURE'] ?? '';
$want = hash_hmac('sha256', $raw, CRM_WEBHOOK_SECRET);

if (!hash_equals($want, $sig)) {
    status_header(401);
    exit;
}

Sign the raw bytes, not a re-encoded copy of the decoded array; key order and whitespace shift and the signature will never match. Store the event id and ignore repeats so a redelivered callback does not apply twice.

Field mapping is where leads quietly die

A field the CRM treats as required and your form treats as optional is a 400 on every submission that leaves it blank. Write the mapping down once, with type and required flag:

Send the internal value for a picklist, not the label, and check the list before adding an option to the form. Empty string and null are not the same thing to every API — pick one and stay consistent.

What to check on a setup that already exists

Submit a test entry with the CRM unreachable and see whether a row lands in the queue. Submit the same entry twice and count the records. Find a row in dead and ask whether anyone was told. Read a week of last_error values — expired tokens, a renamed picklist value, a rate limit nobody raised all show up there as patterns.

The visitor's side of all this is one line of text. Everything that keeps that line honest happens after they close the tab.

If you are building this on WordPress and would rather not maintain the queue and the replay yourself, that is the kind of work we take on: custom development.

What a robots.txt group actually applies to

A crawler fetches the file, matches its own product token against the User-agent lines, and picks one group: the most specific match it can find. Nothing else in the file applies to it, and the order the groups appear in changes nothing.

Two things follow from that, and both surprise people the first time.

A crawler that is not named anywhere falls back to the * group. If the file has no * group at all, that crawler is allowed everywhere. Being absent from the file is not a block.

A crawler that is named exactly gets that group and nothing else. Write a User-agent: X group containing Allow: / and the * group's Disallow: /admin/ no longer reaches X. The named group wins outright, whatever the fallback says.

So the shape for "turn one away, leave the others on their normal terms" is one narrow group plus the fallback that describes everyday access:

# normal rules for everyone
User-agent: *
Disallow: /admin/
Disallow: /search
Disallow: /*?s=

# one crawler, fully out
User-agent: GPTBot
Disallow: /

Allow inside the named group is only needed when a Disallow in that same group would otherwise catch the path — Disallow: / plus Allow: /blog/ keeps the blog open for that crawler. Wildcard and end-anchor matching follows Google's documented syntax; other crawlers implement a subset, so keep expressions simple if the rule has to hold across all of them.

The token matters more than the file. Operators publish separate tokens for separate jobs — a training crawler, a search/answer crawler, a fetcher triggered by a user — and blocking one does not block the others. Google-Extended is not Googlebot: one governs use of your content in Gemini training and grounding, the other governs Search. Block the training token and leave the search token alone, and the page stays in search results. Check the operator's current documentation for the exact token you mean, because names get added and renamed.

Where robots.txt stops

It is a request. A crawler that ignores it loses nothing.

The User-agent line is a string the client writes about itself. Anything can send GPTBot.

No rule applies backwards. Pages already fetched, and anything already in a dataset, stay where they are. A new Disallow is not a deletion.

Blocking the token at the server

nginx, mapping the header to a flag and returning 403:

map $http_user_agent $ai_block {
    default   0;
    ~*GPTBot  1;
}

server {
    listen 443 ssl http2;
    server_name example.com;

    if ($ai_block) {
        return 403;
    }

    # the rest of the site
}

An if wrapping a single return is one of the uses nginx documents as safe. Keep the map at http level and add a line per token — one map, many rules, no repeated if blocks to keep in sync.

Apache, with an environment flag:

SetEnvIfNoCase User-Agent "GPTBot" ai_block=1

<RequireAll>
    Require all granted
    Require not env ai_block
</RequireAll>

Return 403 rather than a 200 with an empty body. A 200 tells the crawler the fetch succeeded and hides the block from your own logs.

Both layers key off the same string and inherit the same weakness: the token can be forged. Server-side blocking raises the effort, not the certainty. It also matches substrings — GPTBot will hit a browser extension that happens to contain the word — so keep the patterns tight and read the 403 log before assuming every hit was the crawler.

Checking that a request is really from the operator

If you would rather allowlist than blocklist, verify the IP instead of trusting the header. Forward-confirmed reverse DNS is the cheap version: look up the PTR record, resolve the hostname forward, and require it to return the original address.

import socket, sys

def fcrdns(ip):
    try:
        host, _, _ = socket.gethostbyaddr(ip)
    except (socket.herror, socket.gaierror):
        return None
    try:
        infos = socket.getaddrinfo(host, None)
    except socket.gaierror:
        return None
    forward = {i[4][0] for i in infos}
    return host if ip in forward else None

for ip in sys.argv[1:]:
    print(ip, fcrdns(ip))

The hostname suffix you accept has to come from the operator's published documentation; a PTR record pointing at a domain you do not recognise should be rejected. Forward confirmation is the step that stops someone pointing their own PTR at a trusted domain and walking through the allowlist.

Worth the trouble only if the rule is allowlist-shaped. For a blocklist, the user-agent map is enough.

An order that saves rework

Decide which token you are blocking, and whether the search token is separate from it. Getting that wrong is the mistake that takes longest to notice.

Add the named group to robots.txt and keep the * group describing everyday access. Honest crawlers honour the file without any server rule, and it documents intent for whoever reads the config next.

Watch the access log before you enforce. You want to know which tokens actually arrive and how often — the AumCrawl plugin tabulates AI crawler hits per bot and lets you allow or block them one at a time, which is quicker than grepping a month of logs. AumCrawl

Then add the server rule and check it with a forged request:

curl -sS -o /dev/null -w '%{http_code}\n' -A 'GPTBot/1.1' https://example.com/

You should see 403 there, and 200 from a normal browser request. Keep the robots.txt rule after the server rule goes live — removing it means the next person to open the file has no idea the block was deliberate.

The blocking is the easy half. Content that was already fetched is the half no config file can undo.

The order below is the point: prove the backup first, measure the restore, and only then book the window.

What "the site" actually includes

Most backup posts stop at the database and wp-content. That archive restores a site that looks fine and quietly loses money.

• the database
• wp-content: uploads, themes, plugins, mu-plugins, and any drop-ins such as object cache or advanced cache files
• wp-config.php, including salts and credentials
• server-level configuration: cron entries, upload and memory limits the application depends on, redirect rules at the proxy or CDN
• TLS certificate issuance and renewal
• DNS records beyond the A record: MX, SPF, DKIM, DMARC, and any verification TXT records
• everything registered elsewhere that points at your domain: payment callbacks, webhook endpoints, OAuth redirect URIs, IP allowlists

That last group is the one that bites. A restore can be perfect and payments still fail, because the provider is still calling the old host.

Frequency is a decision, not a default

• database: daily at minimum, plus one immediately before any plugin or theme update and before any content import
• files: on a schedule matched to how often they change; uploads move constantly on a shop and rarely on a brochure site
• configuration: on change, and in version control
• retention: a few days locally, several weeks off the host
• at least one copy that a compromised admin account cannot delete, with its own credentials and its own access path

The nightly database export, with the day-of-week as the filename so rotation is trivial:

17 2 * * * /usr/local/bin/wp --path=/var/www/site db export /var/backups/db-$(date +\%u).sql --quiet

Note the escaped % — cron treats an unescaped one as a newline and the job fails silently.

Files, plus a checksum you can verify after the copy leaves the machine:

STAMP=$(date +%Y%m%d-%H%M)
tar -czf /var/backups/files-$STAMP.tar.gz -C /var/www/site wp-content wp-config.php
sha256sum /var/backups/*$STAMP* > /var/backups/$STAMP.sha256
chmod 600 /var/backups/*

Encrypt before it goes to shared storage. wp-config.php and the users table travel in that archive, so an unencrypted backup in a bucket is a credential leak with a delay fuse.

The drill: restore somewhere disposable

The restore is the artefact. The archive is only a hypothesis until you have proven it.

wp db create
wp db import /var/backups/db-20261026-0200.sql
wp db check

wp search-replace 'https://www.example.com' 'https://drill.example.net' \
  --all-tables --precise --dry-run

Read the dry run before running it for real. It reports the number of replacements per table. A zero on a table you know contains the domain means the replace is being skipped, not that the table is clean.

wp search-replace 'https://www.example.com' 'https://drill.example.net' \
  --all-tables --precise
wp cache flush

--precise exists because serialised values store their own string lengths. A plain sed or UPDATE ... REPLACE() changes the URL without updating the length prefix and the value becomes unreadable — you find out when a widget renders empty and the settings page refuses to save.

Compare counts between source and drill rather than trusting the import message:

wp db query "SELECT COUNT(*) FROM wp_posts WHERE post_status='publish';"
wp db query "SELECT COUNT(*) FROM wp_postmeta;"
find wp-content/uploads -type f | wc -l

Then walk the checklist on the restored copy, writing down the wall-clock time next to each item:

• home page and one deep page return 200
• log in as an administrator and as a non-administrator user
• an uploaded image renders at full size and as a generated thumbnail (the thumbnail proves the metadata survived, not just the file)
• the contact form submits and the submission arrives where it should
• the scheduled task fires
• outbound mail sends from the right domain and passes SPF and DKIM at the receiving side
• the redirects that matter still work: http to https, non-www to www, old URLs
• site search returns results (a restore often needs its index rebuilt)
• one payment or webhook callback in the provider's sandbox mode

The total at the bottom is your restore time. It is the number the window is built from — not the size of the tarball, and not a guess.

Turning the drill into a window

window = drill time × 1.5 + DNS TTL + verification + rollback budget

The 1.5 accounts for doing it while someone is waiting on you. Lower the DNS TTL to 300 seconds at least one old-TTL period before the window so a rollback flip is fast:

dig +noall +answer www.example.com A

If the arithmetic does not fit what the business can tolerate, do not shorten the verification block. Rehearse again, or split the move: copy files ahead of time, change the database the night of.

The order on the night

The order matters more than the speed.

1. Announce the freeze and stop content edits. Someone will be halfway through a page; tell them before, not during.
2. Take the final export while nothing is writing: database and files from the same moment, or you will spend the window reconciling a mismatch.
3. Import on the target, run the search-replace, fix ownership and permissions.
4. Verify against the target using a hosts-file entry or a temporary hostname, before DNS moves, so a failure never reaches visitors.
5. Flip DNS.
6. Verify again over the public name — the same checklist, because a hostname that resolves is not a site that works.
7. Leave the old host untouched and serving. Do not delete anything yet.

Step 4 looks like this on your own machine:

203.0.113.10  www.example.com

That address is a placeholder; put the target server's real one there, flush the local resolver cache, and test with the production hostname so URL generation, cookies and redirects behave exactly as they will after the flip.

Write the rollback triggers down before you start: checkout failing, logins failing, or an error rate above a threshold you name in advance. And note that anything created on the new host after cutover is lost if you flip back — so nobody edits content until verification passes.

After the window

• keep the old host's data for at least one full backup cycle of the new host
• confirm the new host's own backup job ran, and verify the checksum of the copy that landed offsite
• update everything that pointed at the old location: provider callbacks, uptime monitoring, external cron, allowlists
• watch the error log and the mail queue for the first day, not the first hour

A migration is finished when the new host has produced a backup you have restored once. Until then you have moved a site and inherited the same unproven archive you started with.

A note on where this comes from: AumCreate deploys WordPress sites onto client-owned hosting — that is the Setup service — and this drill is the handover step that makes a go-live date predictable. If you are moving an existing site rather than starting one, the checklist above applies as written.

Two separate jobs got skipped. Sitemaps and robots rules decide whether a crawler can reach a URL and how often it comes back. Meta, headings and structured data decide what it takes away once it arrives. Plenty of sites do one and not the other.

Reaching the page

A single allow-all group covers every crawler that doesn't have its own group:

User-agent: *
Allow: /
Disallow: /cart
Disallow: /checkout

Sitemap: https://example.com/sitemap.xml

Per-bot groups are worth writing only when the rules differ. User-agent: GPTBot followed by Allow: / changes nothing when * already allows everything — it just leaves a file that is harder to audit six months later. Add groups when you block one crawler and let another through.

Two failures in this area cause more missing pages than any block: a CDN or WAF rule that rejects unfamiliar user agents before robots.txt is fetched at all, and a staging Disallow: / that survives the go-live.

Sitemap: a URL list with a date

priority and changefreq are ignored by every consumer that matters. What is left is loc and lastmod, and lastmod only helps while it stays honest.

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/setup</loc>
    <lastmod>2026-10-14</lastmod>
  </url>
  <url>
    <loc>https://example.com/work/home-party-hibachi</loc>
    <lastmod>2026-09-30</lastmod>
  </url>
</urlset>

Constraints worth enforcing:

• absolute URLs, same host as the canonical tag
• no 301s, no 404s, no noindex targets — an entry pointing at a noindex page is a contradiction and burns crawl budget
• beyond 50,000 URLs or 50MB, split by section and list the parts in a sitemap index
• separate sitemaps for pages and for images or products; don't mix types in one file

WordPress serves one at /wp-sitemap.xml. Two changes are usually worth making. Drop the users provider, which publishes author archives you probably don't want indexed:

add_filter( 'wp_sitemaps_add_provider', function ( $provider, $name ) {
    if ( 'users' === $name ) {
        return false;
    }
    return $provider;
}, 10, 2 );

And keep search results and 404s out of the index:

add_filter( 'wp_robots', function ( array $robots ) {
    if ( is_search() || is_404() ) {
        $robots['noindex'] = true;
        $robots['follow']  = true;
    }
    return $robots;
} );

Check that every entry returns 200

Reading a sitemap by eye stops working at about ten URLs. Loop it:

curl -s https://example.com/sitemap.xml \
  | grep -o '<loc>[^<]*</loc>' \
  | sed 's/<[^>]*>//g' \
  | while read -r u; do
      printf '%s %s\n' "$(curl -s -o /dev/null -w '%{http_code}' "$u")" "$u"
    done

Anything that isn't 200 gets fixed or removed. So does a URL that returns 200 but canonicalises elsewhere — the sitemap should hold final URLs only.

Page meta: the frame the summary sits in

Assistants summarise visible text far more than meta tags, but the title and description still set the frame. Generic values produce a generic summary, and the model fills the gaps itself.

<title>WordPress setup: your host, your content, live in 5 working days</title>
<meta name="description" content="We install the template on your own hosting, replace every page with your content, and hand over a one-page editing guide. Live 5 working days after materials arrive.">
<link rel="canonical" href="https://example.com/setup">

A description written as a claim plus specifics survives paraphrasing. "We provide professional web solutions" gives a parser nothing to work with.

Also on every page worth quoting:

• one <h1> describing the page's job, not the site name
• a first paragraph that states the subject without throat-clearing
• og:title and og:description mirroring the title and description, since chat products and link previews read those

Structured data that answers what this is

JSON-LD is the block most likely to be quoted verbatim. Keep it to types that describe the page accurately; an inflated Service block does more damage than no block at all.

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "Service",
  "name": "WordPress template setup",
  "serviceType": "Website setup",
  "provider": {
    "@type": "Organization",
    "name": "Example Pte Ltd",
    "url": "https://example.com/"
  },
  "areaServed": "SG",
  "offers": {
    "@type": "Offer",
    "price": "399",
    "priceCurrency": "USD",
    "url": "https://example.com/setup"
  }
}
</script>

Every value here should also appear in the body. Parsers cross-check against the rendered page; a price in schema that the page never states is the quickest way to get the whole block discounted.

A plain-text map

llms.txt at the site root is a Markdown file listing the pages worth reading, one line each with a link and a description:

# Example Pte Ltd

> WordPress site setup on the client's own hosting, based in Singapore.

## Pages
- [Setup](https://example.com/setup): what is included, what the client supplies, timeline
- [Work](https://example.com/work): delivered projects with links
- [Templates](https://example.com/templates/corporate): demo sites, not client projects

Keep the descriptions factual. It is a table of contents, not a pitch.

Maintaining sitemap, meta, schema and llms.txt as four separate things is where most of this rots. AumViso covers all four from one screen on WordPress, if that is the stack you are on.

What the crawler sees, versus what the browser shows

This sits behind most reports of an assistant being unable to read a site. The browser executes JavaScript; a crawler may not.

curl -s -A GPTBot https://example.com/setup | grep -o '<h1[^>]*>[^<]*</h1>'

An empty result means the main text is client-rendered and the parser receives a shell. The usual cause is a builder that hydrates everything from JSON. Server-render or pre-render at minimum the headings, paragraphs, prices and addresses.

Two quieter failures:

• paragraphs baked into images. Alt text serves screen readers; it does not carry a paragraph into a summary.
• content behind a cookie wall, a login, or an initial-HTML loading placeholder that never resolves for a non-JS fetch.

Order to do it in

Robots and sitemap first — no other fix matters if the URL can't be reached. Then canonical consistency and the 200 check. Then titles, descriptions and headings on the pages you want quoted. Then JSON-LD, then llms.txt. None of it needs a rebuild; the checks are the part people skip, and they are worth re-running after every deploy.

What follows turns that into a decision you can defend: get the string into the log, count it, verify the claim, look at frequency rather than presence, then enforce at the layer that actually enforces.

Get the user agent into the log, with the real client IP

nginx's default combined format already carries $http_user_agent. If the site runs a custom format, add it.

log_format withua '$remote_addr - $remote_user [$time_local] '
                  '"$request" $status $body_bytes_sent '
                  '"$http_referer" "$http_user_agent" '
                  'xff="$http_x_forwarded_for" rt=$request_time';

access_log /var/log/nginx/access.log withua;

Apache's combined nickname includes %{User-Agent}i for the same reason. Confirm with one live request:

tail -n 1 /var/log/nginx/access.log

Behind a CDN or a load balancer, $remote_addr is the edge node, not the crawler — and every verification step below will then check the wrong address. Log $http_x_forwarded_for or the CDN's client-IP header as well, and use that column for the address, remembering the header is only trustworthy when the request came through your own proxy.

Retention matters more than people expect. Seven days of logs cannot separate a crawler that fetches everything hourly from one that fetches everything weekly.

Count the strings first

Two commands get you most of the way before you write any code.

zcat -f /var/log/nginx/access.log* \
  | awk -F'"' 'NF >= 6 { print $6 }' \
  | sort | uniq -c | sort -rn | head -40

The -F'"' split works because the combined format puts the user agent between the fifth and sixth quote, and $6 is the string. NF >= 6 skips malformed lines, which show up whenever a UA itself contains a quote.

The same census per day:

zcat -f /var/log/nginx/access.log* \
  | awk -F'"' 'NF >= 6 { split($1, a, "["); split(a[2], t, ":"); print t[1] "|" $6 }' \
  | sort | uniq -c | sort -rn | head -40

t[1] is the date field of the timestamp. What you get is hits per UA per day, which is enough to tell a crawler that appeared once from one that has been running for a week.

Verify the claim before you act on it

Reverse DNS with a forward confirmation is the cheapest test that means anything. Look up the PTR record for the address, resolve that hostname forward, and check the original IP is in the answer. A PTR record alone proves nothing; anyone can point one at a name that looks official.

The script below reads a log file and prints one row per UA string: hits, days seen, hits per day, distinct addresses, how many of a five-address sample passed the forward-confirmed check, the status mix, and megabytes served.

#!/usr/bin/env python3
"""UA census with reverse-DNS confirmation.

Usage: python3 ua_census.py /var/log/nginx/access.log > census.csv
"""
import csv
import re
import socket
import sys
from collections import defaultdict

LINE = re.compile(
    r'^(?P<ip>\S+) \S+ \S+ \[(?P<ts>[^\]]+)\] '
    r'"(?P<req>[^"]*)" (?P<status>\d{3}) (?P<size>\S+) '
    r'"[^"]*" "(?P<ua>[^"]*)"'
)

VERIFIED = {}


def confirmed_host(ip):
    """Return the PTR host only if a forward lookup resolves back to the same IP."""
    if ip in VERIFIED:
        return VERIFIED[ip]
    host = ""
    try:
        ptr = socket.gethostbyaddr(ip)[0]
        family = socket.AF_INET6 if ":" in ip else socket.AF_INET
        addrs = {info[4][0] for info in socket.getaddrinfo(ptr, None, family)}
        if ip in addrs:
            host = ptr
    except (socket.herror, socket.gaierror, OSError):
        host = ""
    VERIFIED[ip] = host
    return host


def main(path):
    stats = defaultdict(lambda: {
        "hits": 0, "ips": set(), "ok": 0, "bad": 0, "bytes": 0, "days": set(),
    })
    with open(path, encoding="utf-8", errors="replace") as fh:
        for line in fh:
            m = LINE.match(line)
            if not m:
                continue
            ua = m.group("ua") or "-"
            rec = stats[ua]
            rec["hits"] += 1
            rec["ips"].add(m.group("ip"))
            status = m.group("status")
            if status.startswith("2"):
                rec["ok"] += 1
            elif status.startswith("4"):
                rec["bad"] += 1
            try:
                rec["bytes"] += int(m.group("size"))
            except ValueError:
                pass
            rec["days"].add(m.group("ts").split(":")[0])

    out = csv.writer(sys.stdout)
    out.writerow(["ua", "hits", "days", "hits_per_day", "ips",
                  "ptr_ok_sample", "sample_ptr", "pct_2xx", "pct_4xx", "mb"])
    for ua, rec in sorted(stats.items(), key=lambda kv: -kv[1]["hits"]):
        sample = sorted(rec["ips"])[:5]
        ptrs = sorted({h for h in (confirmed_host(ip) for ip in sample) if h})
        days = max(len(rec["days"]), 1)
        out.writerow([
            ua[:120],
            rec["hits"],
            len(rec["days"]),
            round(rec["hits"] / days, 1),
            len(rec["ips"]),
            "{}/{}".format(len(ptrs), len(sample)),
            " ".join(ptrs),
            round(100.0 * rec["ok"] / rec["hits"], 1),
            round(100.0 * rec["bad"] / rec["hits"], 1),
            round(rec["bytes"] / 1048576.0, 1),
        ])


if __name__ == "__main__":
    main(sys.argv[1])

PTR lookups are network calls, so the script caps them at five addresses per UA; a million-line log would otherwise turn into an hour of DNS. Raise the cap over a slice of log if you need a better sample.

Read the columns together. A claimed crawler with ptr_ok_sample at 5/5 and a steady hits_per_day behaves like the thing it names. The same string at 0/5, spread over 40 distinct addresses, is either a spoofer or an operator that publishes no PTR records at all — and both go in the same bucket, unknown client, when you decide.

Frequency is what costs you

A crawler that fetches 200 URLs once a week is a rounding error. One that fetches 20,000 URLs an hour is a different problem. Peak rate per minute, per UA:

zcat -f /var/log/nginx/access.log* \
  | awk -F'"' 'NF >= 6 { split($1, a, "["); split(a[2], t, ":"); print t[1] " " t[2] ":" t[3] "|" $6 }' \
  | sort | uniq -c | sort -rn | head -20

t[2]:t[3] is hour:minute, so the top rows are your peak minutes.

The numbers worth putting next to each other:

• requests per day, and whether that figure is climbing
• distinct URLs per day, since a client hitting the same URL 5000 times is a loop rather than a crawl
• the 4xx share, because a high one usually means the client works from an old URL list and you pay CPU for 404 pages
• megabytes served, which is the part the host bills for
• peak requests per minute, the number that decides whether you need a rate limit at all

Enforce where enforcement exists

robots.txt is a request. Clients that respect it back off; the ones you worry about are defined by not respecting it. Still worth writing, because it costs nothing:

User-agent: GPTBot
Disallow: /

User-agent: *
Allow: /

For actual blocking, do it at nginx or at the CDN edge. A map plus return is the least surprising way in nginx:

map $http_user_agent $blocked_ua {
    default        0;
    "~*GPTBot"     1;
    "~*ClaudeBot"  1;
    "~*CCBot"      1;
}

server {
    ...
    if ($blocked_ua) {
        return 403;
    }
}

return inside if is one of the documented safe uses. Be clear about what it does: it matches a string, so it blocks the honest client and the spoofer together, and it does nothing at all about a fetcher sending a browser UA.

If slowing a burst beats blocking it, rate-limit per address on the PHP location and answer 429 with Retry-After:

limit_req_zone $binary_remote_addr zone=perip:10m rate=120r/m;

server {
    location ~ \.php$ {
        limit_req zone=perip burst=30 nodelay;
        limit_req_status 429;
        add_header Retry-After 60 always;
    }
}

Set the rate above what a normal page load with its assets needs, watch the 429 count for a day, then adjust. Shared corporate NAT addresses will trip a limit set too low, so the 429 log is part of the tuning, not an afterthought.

If parsing logs is not the job you want, AumCrawl records hits per crawler inside WordPress and lets you allow or block each one separately — same decision, counting already done.

Where this falls apart

Tokens that never appear in a UA. Some AI opt-out tokens exist only for robots.txt. Grep for them in an access log and you get zero rows forever, because the fetching is done by the operator's general crawler. Absence from the log is not absence of the crawl.

Quote characters inside the UA. A UA containing " shifts every field after it in a -F'"' split. Run the awk and the Python parser over the same file; if the counts disagree, this is why.

Truncation. Some clients send long strings. Check that the full UA landed in the log rather than the first 100 characters, or you will bucket two different clients as one.

IPv6. gethostbyname only handles IPv4 and fails quietly on v6 addresses. The script uses getaddrinfo with the family chosen from the address itself, which is the part that keeps it honest.

Shared hostnames. A PTR record ending in a cloud provider's domain means the address sits in a data centre, not that it belongs to the operator you have in mind. Treat the domain suffix as a hint and the forward-confirmed match as the fact.

Fetches that look like users. When someone asks an assistant about a page and the assistant reads it live, that request can arrive with an ordinary browser UA from a data-centre address. No UA rule catches it. A referrer or an ASN lookup sometimes hints at it, and neither is reliable enough to act on alone.

Start with the census, confirm a handful of addresses per UA before blocking anything, and keep raw logs long enough that next month's comparison means something.

The built-in connector covers the text. It does not cover the site around the text. Below is the part it can do, the code to drive it from a small plugin, and the places where it stops and you need something else.

What the connector gives a plugin

The connector is a client that ships with recent WordPress builds. It reads a credential the site owner stored — in the database, or from a constant if the host injects one — and exposes a prompt API to plugin code. Plugins call the client; the client talks to whichever provider is connected.

Three consequences matter for a translation job:

• The key is shared. One credential serves the translation plugin, the chatbot installer, the schema plugin, anything else on the site that generates text.
• Your code does not pick the provider per request. Whatever the site owner connected is what answers.
• Nothing about languages, URLs or alternates is included. You get generated text back.

If nothing is connected, plugins calling the client fail, and so will the code below. That is the first thing to check.

Check the site before writing anything

wp eval 'var_dump( function_exists( "wp_ai_client_prompt" ) );'

A true means the client is loaded. Then a real call, which tells you whether a usable credential is behind it:

wp eval '
if ( ! function_exists( "wp_ai_client_prompt" ) ) { echo "no AI client", PHP_EOL; return; }
$r = wp_ai_client_prompt( "Reply with the single word ok" )->generate_text();
if ( is_object( $r ) && method_exists( $r, "get_text" ) ) { echo $r->get_text(), PHP_EOL; }
elseif ( is_wp_error( $r ) ) { echo $r->get_error_message(), PHP_EOL; }
else { echo (string) $r, PHP_EOL; }
'

An error about a missing provider means the client is present and the credential is not usable. Fix that in the admin before going further; there is no plugin-side workaround that does not involve a second key.

The translation call

Drop this in a small plugin or an mu-plugin. No settings page, no key field.

<?php
defined( 'ABSPATH' ) || exit;

function aum_ai_ready(): bool {
    return function_exists( 'wp_ai_client_prompt' );
}

function aum_translate_html( string $html, string $lang, string $glossary = '' ): string {
    if ( ! aum_ai_ready() ) {
        throw new RuntimeException( 'No AI client available on this site.' );
    }

    $prompt = implode( PHP_EOL, array_filter( [
        "Translate the HTML below into {$lang}.",
        'Keep every tag, attribute, shortcode and URL byte-identical.',
        'Do not translate product codes, model numbers, or any string listed in the glossary.',
        'Return the translated HTML only, with no commentary and no code fence.',
        $glossary !== '' ? 'Glossary:' . PHP_EOL . $glossary : '',
        '---',
        $html,
    ] ) );

    $builder = wp_ai_client_prompt( $prompt );
    if ( method_exists( $builder, 'using_temperature' ) ) {
        $builder = $builder->using_temperature( 0.1 );
    }

    $out = $builder->generate_text();

    if ( is_wp_error( $out ) ) {
        throw new RuntimeException( $out->get_error_message() );
    }
    if ( is_object( $out ) && method_exists( $out, 'get_text' ) ) {
        $out = $out->get_text();
    }

    return trim( (string) $out );
}

Why it looks like this:

• using_temperature and get_text are guarded with method_exists. The result object differs between client builds, and the guard costs nothing versus a fatal on a site carrying a different one.
• Temperature is low on purpose. You want consistent terminology across 300 posts, not variety.
• The prompt states the invariants — tags, shortcodes, URLs — because a model will otherwise helpfully rewrite a link or drop a <br>.
• The glossary is where product names, model codes and legal terms survive verbatim. Keep it in a file rather than the database so it survives a migration.

Chunk before you send

Long posts get truncated, and truncated output looks like a model failure. Split on block boundaries and send pieces you can rejoin.

function aum_chunks( string $html, int $limit = 4000 ): array {
    $parts  = preg_split( '/(?<=<\/p>|<\/li>|<\/h2>)/i', $html ) ?: [ $html ];
    $chunks = [];
    $buf    = '';

    foreach ( $parts as $part ) {
        if ( $buf !== '' && strlen( $buf ) + strlen( $part ) > $limit ) {
            $chunks[] = $buf;
            $buf      = '';
        }
        $buf .= $part;
    }
    if ( trim( $buf ) !== '' ) {
        $chunks[] = $buf;
    }

    return $chunks;
}

strlen counts bytes, so 4000 bytes is roughly 4000 Latin characters and considerably fewer Chinese or Japanese ones. Set the limit conservatively and measure against the largest post on the site.

Store with a hash, never on render

function aum_translate_post( int $post_id, string $lang ): string {
    $post = get_post( $post_id );
    if ( ! $post instanceof WP_Post ) {
        throw new RuntimeException( "No post {$post_id}" );
    }

    $store = (array) get_post_meta( $post_id, '_aum_i18n', true );
    $hash  = md5( $post->post_title . '|' . $post->post_content . '|' . $lang );

    if ( isset( $store[ $lang ]['hash'] ) && $store[ $lang ]['hash'] === $hash ) {
        return (string) $store[ $lang ]['content'];
    }

    $content = '';
    foreach ( aum_chunks( $post->post_content ) as $chunk ) {
        $content .= aum_translate_html( $chunk, $lang );
    }

    $store[ $lang ] = [
        'hash'    => $hash,
        'title'   => aum_translate_html( $post->post_title, $lang ),
        'content' => $content,
        'at'      => time(),
    ];
    update_post_meta( $post_id, '_aum_i18n', $store );

    return $content;
}

The hash is the point. Editing one paragraph invalidates that language and nothing else; re-running the command over 400 posts after a wording change on the homepage costs one post worth of tokens.

Two rules for the storage:

• Never call the model inside the_content. A page render that waits on a network call adds seconds, and a traffic spike adds a bill.
• Keep the source hash next to the translation so a later run can tell stale from current without diffing text.

Check the shape of what came back

Models drop an <li> or eat a shortcode often enough that you should verify before saving.

function aum_shape_ok( string $src, string $out ): bool {
    foreach ( [ '<a ', '<h2', '<h3', '<img ', '<ul', '<table' ] as $tag ) {
        if ( substr_count( $src, $tag ) !== substr_count( $out, $tag ) ) {
            return false;
        }
    }

    $grab = static function ( string $s ): array {
        preg_match_all( '/\[[a-z0-9_-]+/i', $s, $m );
        $found = array_values( array_unique( $m[0] ) );
        sort( $found );
        return $found;
    };

    return $grab( $src ) === $grab( $out );
}

Retry once with the same prompt plus a note about which tags went missing. If it fails again, keep the source text for that chunk and log the post ID — a mixed-language page is worse than one that needs a manual pass.

Run it from the command line

if ( defined( 'WP_CLI' ) && WP_CLI ) {
    WP_CLI::add_command( 'aum-translate', function ( array $args ): void {
        [ $post_id, $lang ] = $args;
        $html = aum_translate_post( (int) $post_id, $lang );
        WP_CLI::success( sprintf( '%d characters stored for %s', strlen( $html ), $lang ) );
    } );
}

One post first, then read the stored meta before trusting anything:

wp aum-translate 123 de
wp post meta get 123 _aum_i18n --format=json

Then a batch, with a pause between jobs. Action Scheduler if some plugin already registers it, wp_schedule_single_event otherwise, or a shell loop over post IDs with sleep 2. The point is that nothing sends 400 requests in one go.

Where the connector stops

Everything below sits outside the credential you just reused:

• URLs. Translated content needs its own path, and the connector has no opinion about what it should be.
• hreflang alternates and a sitemap that lists the translated URLs.
• A language switcher, translated menus, translated widgets.
• Slugs. Do not take a slug from model output; generate it with sanitize_title() on the translated title and keep a mapping of old path to new path so inbound links keep working.
• Theme and plugin string translation. The .po/.mo layer is a different mechanism entirely.
• Structured data. Translate the values in a JSON-LD block and leave keys, @type and url untouched; a translated @type breaks the markup silently.
• Shop fields: variation labels, attribute terms, checkout strings.
• Review. Nothing in the connector tells you whether a translation is correct.

Turning stored translations into routes, alternates and a switcher is its own layer. Plugins that run on the same connector do that part: AumLang is one, and it also covers product pages in a shop. Writing that layer yourself is reasonable if you have a fixed page set and can map URLs by hand.

The operational limits that bite later

The bill belongs to the site owner. The credential is theirs and tokens are billed to their provider account. Estimate before the first full run: translate one long post, note the character count, multiply by the site.

Retries need a ceiling. Two attempts per chunk, then mark the post failed and move on. A retry loop against a rate-limited provider turns a partial run into a stuck queue.

Output changes between runs. Retranslating months later with a different connected model gives different wording for the same source. If consistency matters more than freshness, trust the hash and only retranslate on content change.

What you send leaves the site. Drafts, internal notes, anything inside the content goes to the provider. Strip shortcodes that render private data, and do not queue unpublished posts without asking.

There is no fallback when the provider is down. The job fails and the queue holds. Log it and let the scheduled run pick it up.

Practical order: verify the client, translate one post, read the meta, run twenty, read those, then run the rest overnight with a delay between requests.

The safe split is display text on one side, matching keys on the other. Product titles and descriptions are display text. Attribute slugs, custom attribute values, variation meta keys, and SKUs are matching keys. Translate the first group. Leave the second group in one canonical language.

Store the translation beside the product

Do not overwrite the original title or description. Store the translated strings in post meta keyed by language:

update_post_meta(123, '_aum_i18n', [
    'de' => [
        'title' => 'Edelstahl-Trinkflasche 750 ml',
        'short_description' => 'Doppelwandig isoliert, hält 24 Stunden kalt.',
        'description' => '...',
    ],
]);

For attribute labels and option names, a site option is enough when the same labels are reused across products:

update_option('aum_attribute_labels', [
    'de' => [
        'pa_color' => 'Farbe',
        'pa_size' => 'Größe',
        'color' => 'Farbe',
        'size' => 'Größe',
    ],
]);

update_option('aum_variation_option_names', [
    'de' => [
        'Blue' => 'Blau',
        'Red' => 'Rot',
        'Small' => 'Klein',
        'Large' => 'Groß',
    ],
]);

The keys must match the stored label or option name exactly. If your attribute is global, the label passed to the filter is usually the taxonomy name, such as pa_color. For a custom product attribute, it is the array key from _product_attributes, such as color.

Make the product getters return the translation

Put this in wp-content/mu-plugins/woo-product-display-i18n.php. It translates the display fields and leaves the variation keys alone.

<?php
/**
 * Plugin Name: Woo Product Display i18n
 * Description: Translate WooCommerce display text without touching variation keys.
 */

function aum_lang(): string {
    $lang = isset($_COOKIE['aum_lang']) ? sanitize_key($_COOKIE['aum_lang']) : 'en';
    return in_array($lang, ['en', 'de'], true) ? $lang : 'en';
}

function aum_product_text(int $product_id, string $field, string $fallback): string {
    $lang = aum_lang();
    if ($lang === 'en') {
        return $fallback;
    }

    $data = get_post_meta($product_id, '_aum_i18n', true);
    if (!is_array($data) || empty($data[$lang][$field])) {
        return $fallback;
    }

    return (string) $data[$lang][$field];
}

add_filter('woocommerce_product_get_name', function ($name, $product) {
    if (!$product instanceof WC_Product) {
        return $name;
    }
    return aum_product_text($product->get_id(), 'title', $name);
}, 10, 2);

add_filter('woocommerce_product_get_short_description', function ($text, $product) {
    if (!$product instanceof WC_Product) {
        return $text;
    }
    return aum_product_text($product->get_id(), 'short_description', $text);
}, 10, 2);

add_filter('woocommerce_product_get_description', function ($text, $product) {
    if (!$product instanceof WC_Product) {
        return $text;
    }
    return aum_product_text($product->get_id(), 'description', $text);
}, 10, 2);

add_filter('the_title', function ($title, $post_id) {
    if (is_admin() || get_post_type($post_id) !== 'product') {
        return $title;
    }
    return aum_product_text((int) $post_id, 'title', $title);
}, 10, 2);

add_filter('woocommerce_attribute_label', function ($label, $name, $product = null) {
    $lang = aum_lang();
    if ($lang === 'en') {
        return $label;
    }

    $labels = get_option('aum_attribute_labels', []);
    return isset($labels[$lang][$name]) ? (string) $labels[$lang][$name] : $label;
}, 10, 3);

add_filter('woocommerce_variation_option_name', function ($name) {
    $lang = aum_lang();
    if ($lang === 'en') {
        return $name;
    }

    $names = get_option('aum_variation_option_names', []);
    return isset($names[$lang][$name]) ? (string) $names[$lang][$name] : $name;
});

aum_lang() reads a cookie in this example. Replace that function with your language router if the site already sets a current language. The important part is that the filters change output strings only. They do not change the values WooCommerce uses to match a variation.

The the_title filter is there because the default WooCommerce product loop calls get_the_title(), not the product getter. It is guarded to product post type and the front-end. If a theme uses the product getter everywhere, you can leave it out.

Attribute labels and option names

A global attribute has a taxonomy such as pa_color. The term slug is the matching key:

term name: Blue
term slug: blue
variation meta: attribute_pa_color = blue

Translating the term name to Blau is safe if the slug stays blue. Translating the slug to blau is not. The variation meta still says blue, the form posts blau, and the match fails.

A custom product attribute is stored in _product_attributes:

'color' => [
    'name' => 'Color',
    'value' => 'Blue | Red',
    'position' => 0,
    'is_visible' => 1,
    'is_variation' => 1,
    'is_taxonomy' => 0,
],

The variation meta for that attribute is:

attribute_color = Blue

If you change Blue to Blau in the value string, WooCommerce will render Blau as the option value and post Blau. The variation meta still says Blue, so the match fails. The filter above leaves value untouched and only changes the option label.

If you use layered navigation or attribute archive pages, those widgets may read term names directly. The filter above covers the product page dropdown. A term display filter is a separate piece; if you add one, return a cloned term with a changed name, not a changed slug, and keep it out of admin.

Variations stay as they are

Keep one variation record for each combination. Do not create a German variation with attribute_pa_color = blau while the product page sends blue. Do not rewrite attribute_* meta during a translation import. Do not translate _sku; it is an inventory key.

WooCommerce builds the variation label from the product name and attribute labels. Since those are filtered, the customer sees the translated label without any change to the variation post. If you use a variation description, store the translated text in meta and filter the variation description getter; do not overwrite _variation_description with the translated string.

A separate product record per language is a different architecture. It can give you separate URLs, but then you own variation duplication, stock sync, price sync, and order routing. For a small catalog, one product record with translated display fields is easier to keep consistent. If you need both languages indexed, add a language router and hreflang; the meta and filter layer above does not create URLs by itself.

Check the switch before you take an order

• Switch to the translated language on a variable product. The dropdown labels should change, but the option values in the page source should stay blue or Blue.
• Add each variation to the cart. The cart line should show the translated product name and the correct variation. The variation ID and SKU should match the original.
• Change quantity and submit the variation form again. WooCommerce should still find the same variation.
• Place a test order. Check the order admin screen and the order email for the correct item and variation.
• Check the product archive title and the single product tabs.
• If you use layered navigation, check the filter labels separately.

If you do not want to maintain the meta and filters yourself, AumLang is our free WordPress plugin for AI translation, hreflang, and WooCommerce product-page translation: AumLang.

Almost all of it is decided before the file reaches the importer.

Encoding first, because it corrupts silently

The file has to be UTF-8. Excel on a Windows machine saves 'CSV (Comma delimited)' in the system code page, which turns every non-ASCII character into mojibake that survives the import and lands in the database.

file -i products.csv

If it reports charset=iso-8859-1 or unknown-8bit, convert rather than re-exporting by hand:

iconv -f WINDOWS-1252 -t UTF-8 products.csv > products-fixed.csv

A UTF-8 BOM is worse than useless here: it attaches itself to the first header name, so the importer never finds that column and quietly skips it. Strip it:

sed -i '1s/^\xEF\xBB\xBF//' products-fixed.csv

If you are producing the file in a spreadsheet, save through 'Text CSV' and set the character set to UTF-8 explicitly. While you are in there, watch for the other spreadsheet habits that break product data:

• SKUs with leading zeros, which come back as 1234 instead of 001234 unless the column is formatted as text
• Long numeric IDs turned into 1.23457E+14
• Values like 1/2 auto-converted to a date
• Prices typed with a comma decimal separator, or with a currency symbol still attached
• Descriptions pasted from a word processor, carrying curly quotes and stray markup

Prices should be plain decimals with a dot. Stock counts should be integers. An empty cell is not the same as 0, and the importer will treat them differently.

Check the structure with a script, not with your eyes

A CSV with quoted fields cannot be validated by counting commas. This script checks the parts that cause silent partial imports and needs nothing beyond Python 3:

#!/usr/bin/env python3
import csv, sys
from collections import Counter
from urllib.parse import urlparse

path = sys.argv[1]
with open(path, newline='', encoding='utf-8-sig') as fh:
    reader = csv.DictReader(fh)
    header = reader.fieldnames or []
    rows = list(reader)

issues = []
for required in ('Type', 'SKU', 'Name', 'Parent', 'Images'):
    if required not in header:
        issues.append('missing column: ' + required)

for i, row in enumerate(rows, start=2):
    if None in row:
        issues.append(f'line {i}: more fields than the header row')
    for key, value in row.items():
        if key and value and value != value.strip():
            issues.append(f'line {i}: whitespace around {key}')

counts = Counter((r.get('SKU') or '').strip() for r in rows)
for sku, n in counts.items():
    if not sku:
        issues.append('row with an empty SKU')
    elif n > 1:
        issues.append(f'SKU {sku} appears {n} times')

parents = {s for s in counts if s}
for i, row in enumerate(rows, start=2):
    if (row.get('Type') or '').strip() == 'variation':
        parent = (row.get('Parent') or '').strip()
        if parent not in parents:
            issues.append(f'line {i}: parent SKU {parent!r} has no row in this file')

for i, row in enumerate(rows, start=2):
    urls = [u.strip() for u in (row.get('Images') or '').split(',') if u.strip()]
    for url in urls:
        parsed = urlparse(url)
        if parsed.scheme not in ('http', 'https') or not parsed.netloc:
            issues.append(f'line {i}: not an absolute image URL: {url}')

print('\n'.join(issues) if issues else 'no structural problems found')
print(f'{len(rows)} rows, {len(header)} columns')

Duplicate SKUs are the one that hurts most: the importer updates by SKU, so the second row overwrites the first and you end up with fewer products than rows you sent.

Image addresses

The importer downloads images from the URLs in the file, which means the URL has to be reachable from the server doing the import, not from your browser. A CDN with hotlink protection, a staging password, or a firewall that blocks non-browser user agents will all return a page instead of an image, and the row imports without it.

Check from the server:

curl -sI 'https://cdn.example.com/img/a.jpg' | head -n 1
curl -sI 'https://cdn.example.com/img/a.jpg' | grep -i content-type

What the file needs:

• Absolute https:// URLs. /wp-content/uploads/a.jpg is not a URL the importer can fetch.
• Percent-encoded spaces and non-ASCII characters in paths.
• One image per entry, multiple entries separated by commas inside a quoted field.
• No cache-busting query strings on duplicates. The same photo referenced as a.jpg and a.jpg?v=2 becomes two attachments in the media library.

Large catalogs are worth doing in two passes: upload images to the media library first, then import the product rows pointing at the resulting URLs. It turns a long, failure-prone download step into a separate job you can rerun.

Variants are two different kinds of rows in one file

A variable product is a parent row plus one row per variation. Mixing the two up is what produces products that look right on the products screen and have nothing on the front end.

The parent row:

• Type set to variable
• its own SKU
• the attribute name columns, for example Attribute 1 name and Attribute 1 value(s), with all the values that variations will use
• Attribute 1 visible and Attribute 1 global set the way you want them used

Each variation row:

• Type set to variation
• Parent set to the parent's SKU, exactly as written in the parent row
• its own unique SKU
• one attribute value, matching one of the values declared on the parent
• its own price, sale price, stock and weight

The failure to watch for is a variation whose attribute value does not appear in the parent's value list. It either fails to attach or attaches to a variation nobody can select. If the parent declares Small, Medium, Large and a variation says Small , the trailing space is enough to break the match, which is why the validation script above flags whitespace.

Stock belongs on the variation row when variations are managed individually. Setting it only on the parent gives you a product that shows in stock while every variation is sold out.

Per-variation images after the import

The Images column covers the product image. When each variation needs its own photo, the reliable path is to import the catalog first and set the variation images afterwards from a mapping file of SKU to image URL, with the images already in the media library. Upload them:

wp media import ./images/*.jpg --porcelain

Then a variation-images.csv with sku,image_url columns, and this script, run with wp eval-file set-variation-images.php from the WordPress root:

<?php
$handle = fopen(__DIR__ . '/variation-images.csv', 'r');
if (!$handle) {
    WP_CLI::error('cannot open variation-images.csv');
}
fgetcsv($handle); // header
$done = 0;
while (($row = fgetcsv($handle)) !== false) {
    [$sku, $url] = array_map('trim', $row);
    $product_id = wc_get_product_id_by_sku($sku);
    if (!$product_id) {
        WP_CLI::warning("no product for $sku");
        continue;
    }
    $variation = wc_get_product($product_id);
    if (!$variation || !$variation->is_type('variation')) {
        WP_CLI::warning("$sku is not a variation");
        continue;
    }
    $attachment_id = attachment_url_to_postid($url);
    if (!$attachment_id) {
        WP_CLI::warning("image not in the media library: $url");
        continue;
    }
    $variation->set_image_id($attachment_id);
    $variation->save();
    $done++;
}
fclose($handle);
WP_CLI::success("$done variation images set");

attachment_url_to_postid() only resolves URLs that belong to the current site's uploads, so the mapping file has to use the final URLs from this install, not the CDN source.

Cleaning pass before the real run

Work through this on the file itself, then import it into a copy of the site:

• Trim whitespace on SKU, Parent and every attribute column
• Normalise categories to the importer's separator, > between levels and a comma between distinct categories
• Confirm every row has the same number of fields as the header, which the script checks
• Decide what an empty stock cell should mean before the import, not after
• Keep the header row exact: renamed columns are ignored, not guessed
• Import a file of five rows first and open one product, one variation and one image in the browser

Back up before the full run, and split large files so a timeout does not leave you unsure which half landed:

wp db export before-import.sql
awk -v n=200 'NR==1{h=$0; next} {f=sprintf("part_%03d.csv", int((NR-2)/n)); if(!(f in seen)){print h > f; seen[f]=1} print > f}' products.csv

Each part keeps the header row. Import them in order, and if a part fails, you know exactly which SKUs to inspect instead of re-auditing the whole catalog. A working demo store built on the same column layout is at https://aumcreate.com/templates/ecommerce?utm_source=hashnode&utm_medium=content&utm_campaign=hashnode-csv-import-variant-images — it is our own demo, not a client project.

Get the response to describe itself first.

Read the response before touching a cache

for i in 1 2 3; do
  curl -s -o /dev/null \
    -w 'ttfb=%{time_starttransfer} total=%{time_total} bytes=%{size_download}\n' \
    https://example.com/
done

Send it more than once. If a full-page cache is doing its job, the later requests usually differ from the first, because the cache key repeats. If every request sits at the same TTFB with the same byte count, the HTML is being assembled from PHP and the database each time.

Then the headers:

curl -sI https://example.com/ | grep -iE 'cache-control|age|x-cache|vary|etag'

Age above zero means something in front of PHP served a stored copy. X-Cache is not standard, but plenty of setups emit HIT or MISS there. Cache-Control: no-cache with an ETag on the HTML means the browser revalidates each load, which is correct for pages that change.

Run both commands again with a logged-in cookie, and from a machine in another region. Where the numbers diverge is where the layer lives.

Full-page cache: the HTML is rebuilt on every request

This is the nginx version, because the mechanics stay visible. Plugin-based page caches do the same job with less control over the key.

# http context
fastcgi_cache_path /var/cache/nginx/wp levels=1:2 keys_zone=wp:100m
                   inactive=60m max_size=2g use_temp_path=off;

map $http_cookie $wp_bypass_cookie {
    default 0;
    '~*wordpress_logged_in'   1;
    '~*woocommerce_cart_hash' 1;
    '~*wp-postpass'           1;
}

server {
    set $wp_bypass_uri 0;
    if ($request_uri ~* '^/(wp-admin/|wp-json/|cart/|checkout/|my-account/)') {
        set $wp_bypass_uri 1;
    }

    location ~ \.php$ {
        include fastcgi_params;
        fastcgi_pass unix:/run/php/php-fpm.sock;   # match your pool
        fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;

        fastcgi_cache wp;
        fastcgi_cache_key '$scheme$request_method$host$request_uri';
        fastcgi_cache_valid 200 301 302 10m;
        fastcgi_cache_lock on;
        fastcgi_cache_bypass $wp_bypass_cookie $wp_bypass_uri;
        fastcgi_no_cache     $wp_bypass_cookie $wp_bypass_uri;
        add_header X-Cache $upstream_cache_status always;
    }
}

Two things bite people here. The exclusion list has to match the actual paths on the site: /checkout/ in a shop that uses /checkout-2/ is a cache serving a stranger's session page to the next visitor. And add_header is inherited all-or-nothing per level, so if the location block already has one, the server-level headers vanish from that response — keep X-Cache in the same block as the others.

Invalidation is the other half. fastcgi_cache_purge is a third-party module, not something you get for free. Without it, expiry is your only invalidation, so either keep fastcgi_cache_valid short or delete the cache directory from a publish hook:

find /var/cache/nginx/wp -type f -delete

Object cache: fast for anonymous visitors, slow for everyone signed in

A full-page cache is bypassed exactly where logged-in traffic lives, which is where the database work piles up. First confirm a persistent object cache even exists:

ls -l wp-content/object-cache.php

No drop-in, no persistent object cache, whatever a settings screen claims. Install one that talks to Redis or Memcached, then verify it is actually being used instead of trusting the green notice:

redis-cli -h 127.0.0.1 -p 6379 monitor

Load a page in another terminal and watch for read and write commands tied to your key prefix. If the page renders and nothing appears, the drop-in is not connected.

redis-cli info memory | grep used_memory_human
redis-cli config get maxmemory-policy

If Redis is shared with anything that expects keys to survive, an eviction policy that drops keys under pressure will drop those too — WooCommerce sessions among them. That shows up as random logouts, not as a slow page, and it is easy to misread as a caching win.

An object cache repeats expensive queries; it does not make one cheap. A page that takes two seconds to build because of a badly indexed meta query will take two seconds on a cold cache, and the cache will spend its memory holding the result. Look at the query before adding a layer in front of it.

CDN: fast next to the server, slow everywhere else

If TTFB from a machine near the origin is fine and TTFB from another continent is not, the origin is healthy and the distance is the problem. Put the static assets on the edge first — that is the cheapest win and it cannot break sessions. HTML at the edge needs rules: exclude responses that set cookies, respect Vary: Cookie if you cache HTML at all, and purge on deploy.

A CDN in front of a slow origin mostly serves slow pages from more locations. Fix the origin response first, then measure again from the remote host and look for a non-zero Age on a repeat request.

Browser cache: the same files downloaded on every visit

location ~* \.(?:css|js|woff2|png|jpe?g|webp|avif|svg)$ {
    expires 30d;
    access_log off;
}

Long lifetimes are safe for filenames that change when the content changes. WordPress appends ?ver= to enqueued assets, and browsers treat a different query string as a different URL, so the version bump does bust the browser copy even though intermediaries may ignore query strings entirely. Do not set a long lifetime on HTML, and do not set immutable on anything whose URL stays the same across releases.

This layer only helps the second visit. If the complaint is about a first load, it is the wrong place to look.

Order of work

Measure with the curl loop, fix the origin query or PHP work if the object cache shows nothing to cache, then enable the full-page cache with the exclusions, then the object cache for the paths the page cache skips, then the edge, then the asset headers. Run the same curl loop at the end. If the numbers did not move, you changed the wrong layer, and at least you know which one it was not.

More WordPress operations notes live at https://aumcreate.com/blog?utm_source=hashnode&utm_medium=content&utm_campaign=hashnode-cache-layers-which-slow.

The other version of the same mistake is keeping the cart and telling customers to check out and wait for an email. You then get orders sitting in wc-processing with no delivery country, no volume, no application, and a price the customer has decided is final.

An inquiry flow is a different mechanism from a checkout with the payment step removed. It needs a list that survives page loads, fields that map to what a quote actually requires, a stored record that doesn't depend on an email arriving, and one screen where somebody can read the whole thing.

Take the cart out of the loop

WooCommerce will keep offering add-to-cart until you remove it. Kill the template hooks and make products unpurchasable, so nothing can sneak into the cart through a leftover widget or an old link.

<?php
/**
 * Plugin Name: Inquiry list instead of cart
 */

add_action( 'init', function () {
    remove_action( 'woocommerce_after_shop_loop_item', 'woocommerce_template_loop_add_to_cart', 10 );
    remove_action( 'woocommerce_single_product_summary', 'woocommerce_template_single_add_to_cart', 30 );
} );

// Nothing is purchasable, so no code path can add to cart.
add_filter( 'woocommerce_is_purchasable', '__return_false' );

One consequence people miss: with woocommerce_is_purchasable false, the product schema loses its offers block. If you were relying on price in the structured data, it's gone, and that's the correct outcome — you no longer sell at that price directly. Decide whether you still want the product markup at all.

Where the inquiry list lives

The list has to survive navigation without a login. The WooCommerce session is already doing that job for the cart: a cookie identifies the visitor, the contents sit in the database, and it's available on every front-end request.

function aw_inquiry_list() {
    if ( null === WC()->session ) {
        WC()->initialize_session();
    }
    return (array) WC()->session->get( 'aw_inquiry', array() );
}

add_action( 'wp_ajax_aw_inquiry_add', 'aw_inquiry_add' );
add_action( 'wp_ajax_nopriv_aw_inquiry_add', 'aw_inquiry_add' );
function aw_inquiry_add() {
    check_ajax_referer( 'aw_inquiry', 'nonce' );

    $id  = isset( $_POST['id'] ) ? absint( $_POST['id'] ) : 0;
    $qty = isset( $_POST['qty'] ) ? max( 1, absint( $_POST['qty'] ) ) : 1;

    if ( ! $id || 'product' !== get_post_type( $id ) ) {
        wp_send_json_error( array( 'message' => 'Unknown product' ), 400 );
    }

    $list         = aw_inquiry_list();
    $list[ $id ]  = $qty;
    WC()->session->set( 'aw_inquiry', $list );

    wp_send_json_success( array( 'count' => count( $list ) ) );
}

The button replaces the add-to-cart template in both the loop and the single product page:

function aw_inquiry_loop_button() {
    global $product;
    printf(
        '<button type="button" class="button aw-inquiry-add" data-id="%d">%s</button>',
        absint( $product->get_id() ),
        esc_html__( 'Add to inquiry', 'aw' )
    );
}
add_action( 'woocommerce_after_shop_loop_item', 'aw_inquiry_loop_button', 10 );
add_action( 'woocommerce_single_product_summary', 'aw_inquiry_loop_button', 30 );

And the client side, printed in the footer. credentials: 'same-origin' is not optional — without it the session cookie isn't sent on the fetch and every visitor gets an empty list.

add_action( 'wp_footer', function () {
    $config = wp_json_encode( array(
        'ajax'  => admin_url( 'admin-ajax.php' ),
        'nonce' => wp_create_nonce( 'aw_inquiry' ),
    ) );
    ?>
    <script>
    window.awInquiry = <?php echo $config; ?>;
    document.addEventListener('click', function (e) {
      var btn = e.target.closest('.aw-inquiry-add');
      if (!btn) return;
      e.preventDefault();
      var body = new URLSearchParams();
      body.append('action', 'aw_inquiry_add');
      body.append('nonce', window.awInquiry.nonce);
      body.append('id', btn.dataset.id);
      fetch(window.awInquiry.ajax, { method: 'POST', credentials: 'same-origin', body: body })
        .then(function (r) { return r.json(); })
        .then(function (res) {
          if (res.success) { btn.textContent = 'Added'; btn.disabled = true; }
        });
    });
    </script>
    <?php
} );

Fields that map to a quote

Every field you add costs completions. The test for keeping one is whether the salesperson would have to send an email to ask for it before quoting.

Free text for the application beats a dropdown of ten options, because the customer who types three lines about their environment is the customer worth calling back. Keep the reference fields as structured data and let everything else be prose.

Variable products are the awkward case. For an inquiry, store the parent product ID and let the customer describe the variant in the free-text field, or store the chosen attributes as a separate meta value. Do not try to reuse the variation lookup — the customer is asking, not ordering, and they will describe variants you don't have.

Store the record first, send the email second

Email fails. It lands in spam, the relay has a bad afternoon, the salesperson is on leave. If the inquiry only exists as an email, it's gone. Insert a post, then send.

add_action( 'admin_post_nopriv_aw_inquiry_submit', 'aw_inquiry_submit' );
add_action( 'admin_post_aw_inquiry_submit', 'aw_inquiry_submit' );
function aw_inquiry_submit() {
    check_admin_referer( 'aw_inquiry_submit' );

    // Honeypot: a real visitor never fills a hidden field.
    if ( ! empty( $_POST['aw']['website'] ) ) {
        wp_safe_redirect( home_url( '/thank-you/' ) );
        exit;
    }

    $in    = isset( $_POST['aw'] ) && is_array( $_POST['aw'] ) ? wp_unslash( $_POST['aw'] ) : array();
    $field = function ( $key ) use ( $in ) {
        return isset( $in[ $key ] ) ? sanitize_text_field( $in[ $key ] ) : '';
    };

    $email = sanitize_email( $field( 'email' ) );
    if ( ! is_email( $email ) ) {
        wp_die( 'Please enter a valid email address.', 'Inquiry', array( 'response' => 400 ) );
    }

    $qty   = isset( $_POST['qty'] ) && is_array( $_POST['qty'] ) ? array_map( 'absint', wp_unslash( $_POST['qty'] ) ) : array();
    $lines = array();
    foreach ( aw_inquiry_list() as $id => $default ) {
        $product = wc_get_product( $id );
        if ( ! $product ) {
            continue;
        }
        $lines[] = array(
            'id'   => (int) $id,
            'name' => $product->get_name(),
            'sku'  => $product->get_sku(),
            'qty'  => isset( $qty[ $id ] ) ? max( 1, (int) $qty[ $id ] ) : 1,
        );
    }
    if ( ! $lines ) {
        wp_die( 'The inquiry list is empty.', 'Inquiry', array( 'response' => 400 ) );
    }

    $post_id = wp_insert_post( array(
        'post_type'   => 'aw_inquiry',
        'post_status' => 'private',
        'post_title'  => $field( 'company' ) . ' - ' . current_time( 'Y-m-d H:i' ),
        'meta_input'  => array(
            'aw_name'        => $field( 'name' ),
            'aw_company'     => $field( 'company' ),
            'aw_email'       => $email,
            'aw_country'     => $field( 'country' ),
            'aw_application' => $field( 'application' ),
            'aw_volume'      => $field( 'volume' ),
            'aw_needed_by'   => $field( 'needed_by' ),
            'aw_lines'       => $lines,
        ),
    ), true );

    if ( is_wp_error( $post_id ) ) {
        wp_die( 'Could not save the inquiry.', 'Inquiry', array( 'response' => 500 ) );
    }

    $body = sprintf(
        "Name: %s\nCompany: %s\nEmail: %s\nCountry: %s\n\n",
        $field( 'name' ), $field( 'company' ), $email, $field( 'country' )
    );
    foreach ( $lines as $line ) {
        $body .= sprintf( "- %s (%s) x %d\n", $line['name'], $line['sku'], $line['qty'] );
    }
    $body .= sprintf(
        "\nApplication: %s\nVolume: %s\nNeeded by: %s\n\nEdit: %s",
        $field( 'application' ), $field( 'volume' ), $field( 'needed_by' ),
        admin_url( 'post.php?post=' . $post_id . '&action=edit' )
    );

    wp_mail( get_option( 'admin_email' ), 'Inquiry: ' . $field( 'company' ), $body );
    wp_mail( $email, 'We received your inquiry', "Thanks, we will reply within one business day.\n" );

    WC()->session->set( 'aw_inquiry', array() );

    wp_safe_redirect( home_url( '/thank-you/' ) );
    exit;
}

Send from an address on your own domain, not admin_email on a shared host. PHP's mail function leaving from a domain without SPF and DKIM gets filtered often enough that the auto-reply is the first thing to disappear. Route it through an SMTP plugin or your host's relay and send yourself one test before launch.

Reading it in the admin

The record needs a screen. A private custom post type gives you the list view, search, and the permalink you put in the notification email.

add_action( 'init', function () {
    register_post_type( 'aw_inquiry', array(
        'label'     => 'Inquiries',
        'public'    => false,
        'show_ui'   => true,
        'supports'  => array( 'title' ),
        'menu_icon' => 'dashicons-email-alt',
    ) );
} );

add_filter( 'manage_aw_inquiry_posts_columns', function ( $cols ) {
    return array(
        'cb'      => $cols['cb'],
        'title'   => 'Inquiry',
        'company' => 'Company',
        'email'   => 'Email',
        'country' => 'Country',
        'items'   => 'Items',
        'date'    => 'Date',
    );
} );

add_action( 'manage_aw_inquiry_posts_custom_column', function ( $col, $post_id ) {
    if ( 'items' === $col ) {
        $lines = get_post_meta( $post_id, 'aw_lines', true );
        echo is_array( $lines ) ? count( $lines ) : 0;
        return;
    }
    if ( in_array( $col, array( 'company', 'email', 'country' ), true ) ) {
        echo esc_html( get_post_meta( $post_id, 'aw_' . $col, true ) );
    }
}, 10, 2 );

add_action( 'add_meta_boxes', function () {
    add_meta_box( 'aw_inquiry_lines', 'Requested items', function ( $post ) {
        $lines = get_post_meta( $post->ID, 'aw_lines', true );
        if ( ! is_array( $lines ) || ! $lines ) {
            echo '<p>No items.</p>';
            return;
        }
        echo '<table class="widefat"><thead><tr><th>Product</th><th>SKU</th><th>Qty</th></tr></thead><tbody>';
        foreach ( $lines as $line ) {
            printf(
                '<tr><td>%s</td><td>%s</td><td>%d</td></tr>',
                esc_html( $line['name'] ), esc_html( $line['sku'] ), (int) $line['qty']
            );
        }
        echo '</tbody></table>';
    }, 'aw_inquiry', 'normal', 'high' );
} );

The form itself is a shortcode on a normal page, with the list rendered server-side so the customer sees what they picked, quantity fields they can correct, and the honeypot field hidden off-screen rather than with display:none — some bots skip hidden inputs.

What breaks after you ship it

The inquiry page cannot be behind a full-page cache. The list is rendered from the session, and the nonce expires; a cached copy hands the second visitor a stale nonce and the submission bounces with a -1. Exclude that page from caching, and if you use a CDN, exclude it there too.

A 400 on submit with no visible error is almost always the nonce, and the nonce is almost always the cache.

Sessions expire. A customer who builds a list on Monday and submits on Thursday gets an empty inquiry. If the product matters to you, mirror the IDs into a cookie with a long lifetime and merge it back on read, rather than extending the WooCommerce session for everyone.

Quantity fields submitted without matching product IDs fail silently if you don't check get_post_type in the handler — the customer can post any integer they like as a product ID, so treat every ID from the request as untrusted.

Last, decide who reads the record. A private post type that nobody opens is the same as no record at all; either point the notification at a shared mailbox the team actually monitors, or agree on a weekly sweep of the admin list.

If the inquiry flow needs to go past a list and a form — variant configurators, file uploads per line item, routing to different salespeople by country — that is the kind of thing we build as custom work, and the starting point is the services page.

Treating crawler traffic as an attack throws away referrals you may want. The narrower job is to find which crawler is actually expensive, and slow that one down without touching the rest.

You need three things from the log before you change a single line of config: which user agent is responsible, how much origin time it accounts for, and which URLs it keeps hitting. Rate of requests alone tells you very little, because a bot fetching cached HTML is close to free while a bot walking a faceted product filter is not.

Log the fields you need, not the ones that look tidy

The combined log format has bytes and status but no timing, so it cannot answer the question. Add a JSON format that carries $request_time, $upstream_response_time and the user agent. $request_time includes the time spent pushing bytes to a slow client; $upstream_response_time is the part PHP and the database are responsible for, which is what you're chasing.

# http context
log_format bot escape=json '{"ip":"$remote_addr","t":"$time_iso8601","req":"$request","status":$status,"rt":$request_time,"urt":"$upstream_response_time","ua":"$http_user_agent","ref":"$http_referer"}';
access_log /var/log/nginx/bot.json bot;

Reload, let it collect for a full day, and make sure the window covers a busy period. A sample taken over a quiet afternoon will name the wrong crawler.

Total origin seconds per user agent

Request counts sort bots by how chatty they are. Sorting by accumulated upstream time sorts them by how much they cost.

jq -r '[.ua, .urt] | @tsv' /var/log/nginx/bot.json \
  | awk -F'\t' '{ t[$1] += $2; c[$1]++ }
                END { for (u in t) printf "%9.1f  %7d  %8.2f  %s\n", t[u], c[u], t[u]/c[u], u }' \
  | sort -rn | head -20

Columns are total upstream seconds, request count, average seconds per request, and the user agent. Example shape of the output:

   1842.6    18221     0.10  Mozilla/5.0 AppleWebKit/537.36 ... GPTBot/1.1
    301.4      4102     0.07  Mozilla/5.0 ... ClaudeBot/1.0
     88.9     31204     0.00  Mozilla/5.0 ... SomeMonitoringService

That third line is the one people misread. Thirty thousand requests and almost no cost means those requests are being served from cache or returning a fast static response. Leave it alone.

Total for the window, so you know what share the top bot holds:

jq -r '.urt // 0' /var/log/nginx/bot.json | awk '{ s += $1 } END { print s " upstream seconds total" }'

See what the expensive bot is asking for

A high average cost usually has a boring explanation: the bot is walking pagination, filter query strings, or a search endpoint.

jq -r 'select(.ua | test("gptbot"; "i")) | .req' /var/log/nginx/bot.json \
  | sort | uniq -c | sort -rn | head -20

If the top URLs are product pages that render from cache, the cost is somewhere else. If they are /?s=, ?filter_, or ?orderby=, every one of those is a fresh database query and you have found the leak. Those URLs should be blocked to all crawlers in robots.txt regardless of which bot is fetching them, because they are both expensive and useless as an indexed page.

Check what status codes the bot receives while you're there. A crawler that keeps retrying 404s and 500s is generating load for nothing:

jq -r 'select(.ua | test("gptbot"; "i")) | .status' /var/log/nginx/bot.json \
  | sort | uniq -c | sort -rn

Deciding which crawler to limit first

Work down the list from the top of the seconds table, and skip anything in the left column.

The rule in one line: highest total upstream seconds, among the crawlers whose traffic you can afford to slow. That is usually the one at the top of the table that isn't your search engine of record.

Don't try to solve this by adding CPU. The same crawler will find the same slow URLs on the bigger machine.

Rate-limit by user agent, on dynamic locations only

Start with robots.txt. It costs nothing and the well-behaved crawlers honour it, though not all of them honour Crawl-delay specifically. Check the log after a day and see whether the rate actually dropped; the log will tell you which bots read it.

User-agent: Bytespider
Crawl-delay: 10

User-agent: *
Disallow: /?s=
Disallow: /*?filter_

For the ones that ignore it, use nginx. Map the user agent to a key, then rate-limit on that key. Requests with an empty key are not counted against any zone, so every other visitor keeps the normal path.

# http context
map $http_user_agent $botkey {
    default              "";
    ~*gptbot             "gptbot";
    ~*claudebot          "claudebot";
    ~*perplexitybot      "perplexitybot";
    ~*bytespider         "bytespider";
    ~*amazonbot          "amazonbot";
    ~*(googlebot|bingbot) "";
}

limit_req_zone $botkey zone=botrate:10m rate=20r/m;

server {
    location / {
        limit_req        zone=botrate burst=5 nodelay;
        limit_req_status 429;
        # ... your existing fastcgi/php config
    }
}

Twenty requests a minute with a burst of five is an example, not a recommendation. Pick a number just under the current rate of the bot you're slowing, so the change is a step down and not a wall.

Two things to keep in mind. The limit sits in location /, so static assets served from their own locations are unaffected — that's intentional, they were never the problem. And a 429 still costs a connection and a worker slot. It's far cheaper than rendering a filtered product page, but it isn't free, so a very aggressive rate limit with a large burst doesn't help as much as people expect.

If you'd rather block: return 403 is the same mechanism and a bigger decision. When you block a crawler outright, whatever surface it feeds stops including you. Rate limiting keeps you in the index while keeping the server alive, which is usually the trade you want.

Verify, then leave it alone

After a full day, count the throttled responses by user agent:

jq -r 'select(.status == 429) | .ua' /var/log/nginx/bot.json \
  | sort | uniq -c | sort -rn | head

Almost everything in that list should be bots. If human user agents show up, your key is too broad — the usual cause is a regex that matches a browser string, or a mobile carrier proxy that rewrites user agents into something that looks like a bot.

Then re-run the seconds-per-agent query for the same length of window and compare. You're looking for the top line's total to fall while the human traffic lines stay flat.

Two things that will bite you

User agents are self-declared strings. The bot that respects your config will send the real one; a scraper that wants to look like the bot will send the same one. If the rule starts charging a bot's traffic to random scrapers, the fix is not a better regex — it's forward-confirmed reverse DNS on the IP, or an allowlist of the operator's published address ranges, checked before the rate limit.

And make sure your own monitoring and uptime checks are excluded before you turn this on. There is nothing quite like rate-limiting your own health check and discovering it during the next incident.

If you want the per-crawler allow/deny without hand-maintaining the map block, that is what we built AumCrawl for: it records which AI crawlers are hitting the site and lets you allow or block each one separately.

1. Pick the URL shape, then live with what it forces

Two facts worth stating plainly, because they drive most of the cost:

• Subdirectory shapes keep every URL on one hostname. Nothing to provision, and every internal link, sitemap entry and canonical stays on a single origin.
• Subdomain and per-domain shapes require DNS and TLS work per language, plus a mapping layer inside WordPress if you use one network. Budget that before you promise a launch date.

If you have no operational reason to split hostnames, subdirectory is the cheaper shape in both models.

2. What multisite actually shares

Multisite is not "separate WordPress installs". It is one install, one user table, one plugin directory, N content tables.

Shared across the network:

• wp_users, wp_usermeta — one account, roles assigned per site
• wp_blogs, wp_site, wp_sitemeta — the network itself
• wp-content/plugins, wp-content/themes, wp-content/mu-plugins

Per site (N = blog ID):

• wp_N_posts, wp_N_postmeta, wp_N_options, wp_N_terms, wp_N_term_relationships, wp_N_comments
• uploads under wp-content/uploads/sites/N/

Verify it on a network with at least two sites:

# tables that belong to blog 2 only
wp db query "SELECT TABLE_NAME FROM information_schema.TABLES
  WHERE TABLE_SCHEMA = DATABASE() AND TABLE_NAME LIKE 'wp\_2\_%'
  ORDER BY TABLE_NAME;"

# users are shared: same IDs from either site
wp user list --url=https://example.com --fields=ID,user_login,roles
wp user list --url=https://example.com/es --fields=ID,user_login,roles

A translation layer gives you the opposite trade: one wp_posts table containing every language, every query filtered by language, and every template that runs a raw WP_Query is now a place where the wrong language can leak in as a fallback. Neither model is "correct" — but the failure modes are different, and you should pick the one whose failure you can debug at 2am.

The real advantage of multisite is wp_N_options: separate menus, separate front page, separate plugin settings, separate permalink structure per language. If your languages will diverge structurally, that is the deciding factor. If they will stay literal translations of each other, a shared table with a language filter is less machinery.

3. Plugin compatibility: the three failure modes

Before you commit, audit every plugin you intend to keep.

Hardcoded table prefix. Any raw SQL touching wp_ instead of $wpdb->prefix or $wpdb->base_prefix will read and write site 1's data from site 2.

grep -rnE "(FROM|INTO|UPDATE|JOIN)[[:space:]]+wp_" \
  wp-content/plugins/your-plugin --include='*.php'

Zero hits is the expectation. Hits are a blocker, not a warning.

Settings stored in the wrong scope. A license key or API credential written with get_option() becomes per-site, and you re-enter it on every language. That should be get_site_option() / update_site_option().

grep -rn "get_site_option\|update_site_option\|get_option(\|update_option(" \
  wp-content/plugins/your-plugin --include='*.php' | head -40

You are looking for which settings are network-wide in intent. This is a manual read, not a grep verdict.

Tables created only on activation. Custom tables are usually created in an activation hook, which fires once — for the site that activated it. Site 2 gets no table and throws database errors.

# activate the plugin on site 2, load its admin screen, then:
wp db query "SHOW TABLES LIKE '%yourplugin%';"

If only wp_yourplugin... exists and wp_2_yourplugin... is missing, the plugin is not multisite-safe without a workaround.

4. Converting to subdirectory multisite, in order

Do this on staging first. The conversion is not a two-way door.

# 0. backup
wp db export pre-multisite.sql

# 1. deactivate everything (the converter requires it)
wp plugin deactivate --all

# 2. convert — the site must be installed at the domain root
wp core multisite-convert --title="Example Network"

The command prints the constants to paste into wp-config.php. They look like this:

// wp-config.php — above the "That's all, stop editing" line
define( 'WP_ALLOW_MULTISITE', true );
define( 'MULTISITE', true );
define( 'SUBDOMAIN_INSTALL', false );   // true for es.example.com

define( 'DOMAIN_CURRENT_SITE', 'example.com' );
define( 'PATH_CURRENT_SITE', '/' );
define( 'SITE_ID_CURRENT_SITE', 1 );
define( 'BLOG_ID_CURRENT_SITE', 1 );

Then replace the WordPress block in .htaccess with the subdirectory rules:

RewriteEngine On
RewriteRule .* - [E=HTTP_AUTHORIZATION:%{HTTP:Authorization}]
RewriteBase /
RewriteRule ^index\.php$ - [L]

# add a trailing slash to /wp-admin
RewriteRule ^([_0-9a-zA-Z-]+/)?wp-admin$ $1wp-admin/ [R=301,L]

RewriteCond %{REQUEST_FILENAME} -f [OR]
RewriteCond %{REQUEST_FILENAME} -d
RewriteRule ^ - [L]
RewriteRule ^([_0-9a-zA-Z-]+/)?(wp-(content|admin|includes).*) $2 [L]
RewriteRule ^([_0-9a-zA-Z-]+/)?(.*\.php)$ $2 [L]
RewriteRule . index.php [L]

Add the language site and check it responds:

wp site create --slug=es --title="Example ES" --email=admin@example.com
wp option update WPLANG es_ES --url=https://example.com/es
wp rewrite flush --url=https://example.com/es

curl -sI https://example.com/es/ | head -1

Common breakages at this step: the site was not installed at the domain root, a page with the slug es already exists and now shadows the new site, or WPLANG is set before the language files are installed so the site silently stays in English.

5. hreflang across a network

A translation layer usually emits alternates for you. A network does not — each site thinks it is alone. Check first:

curl -sL https://example.com/es/pagina/ | grep -c 'rel="alternate" hreflang='

0 means you are serving translated pages with no cluster at all. Relate translations with a shared meta value, then emit the full set from one place:

<?php
// wp-content/mu-plugins/network-hreflang.php
// Requires a shared _translation_group meta value on each translated post.
add_action( 'wp_head', function () {
    if ( ! is_multisite() || ! is_singular() ) {
        return;
    }

    $post_id = get_queried_object_id();
    $group   = get_post_meta( $post_id, '_translation_group', true );
    if ( ! $group ) {
        return;
    }

    $post_type = get_post_type( $post_id );
    $links     = [];

    foreach ( get_sites( [ 'number' => 0 ] ) as $site ) {
        switch_to_blog( (int) $site->blog_id );

        $found = get_posts( [
            'post_type'      => $post_type,
            'post_status'    => 'publish',
            'meta_key'       => '_translation_group',
            'meta_value'     => $group,
            'posts_per_page' => 1,
            'fields'         => 'ids',
            'no_found_rows'  => true,
        ] );

        if ( $found ) {
            $links[ get_locale() ] = get_permalink( $found[0] );
        }

        restore_current_blog();
    }

    if ( count( $links ) < 2 ) {
        return;
    }

    foreach ( $links as $locale => $url ) {
        printf(
            '<link rel="alternate" hreflang="%s" href="%s" />' . "\n",
            esc_attr( str_replace( '_', '-', $locale ) ),
            esc_url( $url )
        );
    }
}, 5 );

Notes that matter in production:

• The set must include the current page itself, and every target must be a canonical URL. If site 2 redirects www to non-www, emit the final URL, not the configured home_url() blindly.
• switch_to_blog() is a context switch with query cost. With more than a handful of sites, cache $links in a transient keyed by $group and delete it on save_post.
• Verify per page, not once: curl -sL <url> | grep -o 'hreflang="[^"]*"' should list one entry per language that actually has a translation.

6. Maintenance cost, side by side

The backup row is where teams get caught. A network restored from a single-site dump comes back with the main site and nothing else.

7. Decision checklist

Answer these before choosing:

1. Will the languages diverge structurally (different menus, different front pages, different plugins)? → multisite.
2. Is it one product catalog with translated strings? → usually a single site.
3. Do translators need their own admin accounts with scoped roles? → multisite handles this natively.
4. Can you back up and restore N tables and rewrite N sets of siteurl/home? If not, the network will eventually be restored wrong.
5. Do you need to kill a single language quickly without touching the others? → multisite, with the caveat that you lose its content.
6. Do you have DNS and TLS automation for subdomains or separate domains? If not, stay on a subdirectory shape.

If you would rather not run the network yourself, we do WordPress deployment and custom builds where this decision is made up front: https://aumcreate.com/services?utm_source=hashnode&utm_medium=content&utm_campaign=hashnode-multisite-vs-single-multilang-decision

• llms.txt — close to zero functional overlap with an all-in-one SEO plugin. The output is one text file. The open question is whether AI crawlers actually consume it, not whether it collides with something you already run.
• Schema / JSON-LD — heavy overlap. A second plugin that emits structured data is worth it only when you can name the exact entity type your current setup does not emit, and you can switch the equivalent module off in the current one.

Below is the check sequence that turns that into a decision for your site.

What actually overlaps

Two rules fall out of that table:

1. One owner per output. For each row, exactly one plugin is allowed to write it. Keep that list in the repo, not in someone's head.
2. Disable before you enable. Never activate the new plugin first and plan to clean up afterwards.

Three checks, five minutes

Run these against the page with the most schema on it — a product page, or a long article with FAQ blocks.

1. Count and inspect the JSON-LD blocks

curl -sL https://example.com/sample-page/ \
  | python3 -c '
import sys, json, re
html = sys.stdin.read()
blocks = re.findall(r"<script[^>]+application/ld\+json[^>]*>(.*?)</script>", html, re.S)
print(f"{len(blocks)} JSON-LD block(s)")
for b in blocks:
    try:
        data = json.loads(b)
    except json.JSONDecodeError as e:
        print("INVALID:", e); continue
    nodes = data if isinstance(data, list) else data.get("@graph", [data])
    for n in nodes:
        print(" ", n.get("@type"), "|", n.get("@id", ""))
'

How to read the output:

• The same @type twice with different @id values → two entities. This is the case that hurts: an Organization at https://example.com/#organization and another at https://example.com#organization are not merged.
• The same @type twice with the same @id → merged, but check that the properties do not contradict each other (two different logo values, two different telephone).
• INVALID → a plugin is printing unescaped content into the script tag. Fix that before adding anything new.
• More than one block is not automatically wrong. A single @graph and several separate blocks are both fine, as long as no entity is declared twice.

2. Duplicate meta tags

curl -sL https://example.com/sample-page/ \
  | grep -oE '<(meta|link)[^>]*(description|canonical)[^>]*>'

Expect exactly one description and one canonical. Two of either means output ordering is already contested; a third writer makes it worse, not better.

3. Is anything serving llms.txt today

curl -s -o /dev/null -w '%{http_code}\n' https://example.com/llms.txt
curl -s https://example.com/llms.txt | head -3

• 404 → nothing there.
• 200 but the body starts with <!DOCTYPE → your fallback handler is returning 200 for missing files. Treat it as missing; a text file is what crawlers parse.
• Markdown starting with # → already covered, leave it alone.

While you are in there, check whether anything already talks to AI crawlers at all:

curl -s https://example.com/robots.txt \
  | grep -iE 'gptbot|claudebot|perplexitybot|google-extended|ccbot'

Decision rules

Add a second schema plugin only if all three are true:

1. You can name the entity type you are missing (for example Product with Offer variants, Service, Event). "Better schema" is not a gap description.
2. The current plugin lets you disable its schema module — globally, or at least per post type.
3. You have re-fetched the three test pages (home, single post, product/archive) and the entity count went down or stayed equal while the missing type appeared.

Do not add a second schema plugin if: the current one already emits the type, or you cannot disable its module, or the gap is really "my page builder prints its own JSON-LD too". Three emitters is worse than two.

For llms.txt, the decision is not a vendor decision. The proposal is a community convention; no major search engine has committed to parsing it, so treat it as cheap hygiene rather than a ranking lever. Generating the file yourself is a legitimate option:

<?php
// wp-content/mu-plugins/llms-txt.php — serves /llms.txt from a template file
add_action( 'init', function () {
    add_rewrite_rule( '^llms\.txt$', 'index.php?llms_txt=1', 'top' );
} );

add_filter( 'query_vars', function ( $vars ) {
    $vars[] = 'llms_txt';
    return $vars;
} );

add_action( 'template_redirect', function () {
    if ( ! get_query_var( 'llms_txt' ) ) {
        return;
    }
    header( 'Content-Type: text/plain; charset=utf-8' );
    $path = get_stylesheet_directory() . '/llms.txt';
    echo file_exists( $path ) ? file_get_contents( $path ) : '';
    exit;
} );

Flush rewrite rules once after adding it (wp rewrite flush), then confirm with the curl above that you get text/plain, not an HTML page.

If you do add a second output plugin, order matters

# 1. snapshot what the site emits today
curl -sL https://example.com/sample-page/ \
  | grep -oE '<(meta|link|script)[^>]*>' | sort > before.txt

# 2. disable the overlapping module in the EXISTING plugin first
#    (its schema module / its sitemap module) — one at a time

# 3. re-fetch and diff
curl -sL https://example.com/sample-page/ \
  | grep -oE '<(meta|link|script)[^>]*>' | sort > after.txt
diff before.txt after.txt

If step 3 shows the schema disappearing entirely, the existing plugin was the only owner — good, now the new one can take over. If it shows more tags than before, stop: something is double-emitting.

Maintenance cost, plainly

• Every added plugin is another update surface and another place where a core or plugin release can change output ordering.
• Two Organization nodes with mismatched @id values is the most common silent breakage, and it is invisible in the admin.
• Two sitemaps means two submissions; pick one and remove the other.
• Re-running the three checks above takes about three minutes. Put it on a monthly calendar rather than trusting that nothing changed.
• Record the ownership table from the top of this post next to the site's deploy notes, so the next person does not "just add another plugin".

Short verdict list

• Suite covers the schema you need, you only want llms.txt → do not add a second schema plugin; ship the file.
• Suite emits generic schema, you need a specific type it cannot produce → add one, and disable the suite's schema module for that post type.
• Page builder also emits JSON-LD → count first, then remove one emitter; the number of plugins is not the metric.
• You want to know which AI crawlers actually request pages on your site → that is a log question, not a schema or llms.txt question.

We build AumViso (free, on wordpress.org), which covers llms.txt, a schema graph, sitemap, meta and internal links — same category as the plugins discussed above, so judge it with the checks in this post rather than with a feature list: https://aumcreate.com/plugins/aumviso?utm_source=hashnode&utm_medium=content&utm_campaign=hashnode-seo-suite-llms-schema-overlap

If your actual question is "which bots are hitting me and should I let them", that is a separate piece of plumbing: https://aumcreate.com/crawlers?utm_source=hashnode&utm_medium=content&utm_campaign=hashnode-seo-suite-llms-schema-overlap

Here is what each one has to answer, what goes on it, and how to tell it is done.

Page 1 — Home

The question it answers: What is this, and is it for me?

What goes on it:

• One sentence, plain language, saying what you sell and to whom. Not a slogan. "We install commercial kitchen extraction systems for restaurants in the west of the country" is a sentence. "Engineering comfort" is not.
• Three or four lines on what makes the offer concrete: what you actually deliver, in what timeframe, for what kind of customer.
• Proof that a stranger can check — a real project photo, a client name you have permission to use, a location, a certification number.
• One obvious next step, repeated at the top and the bottom.

What kills it: a carousel nobody watches, a stock photo of people shaking hands, no address, no indication of which country you are in, and a hero section that says nothing a competitor could not also say.

Done test: show it to someone outside your industry for ten seconds and ask what the company does. If they cannot say it back in one sentence, rewrite the sentence, not the design.

Page 2 — What you sell

The question: What exactly am I buying, and in which version?

One page if you sell one thing. An index page plus a page per line if you sell several. Each item needs:

• what it is, in the customer's words, not the internal product name
• who it is for and who it is not for
• what is included, and — this matters more than most people think — what is explicitly not included
• the variants, sizes, models, or service tiers, as a list or a table
• one photo per variant, on a plain background, named in plain language

Done test: a customer can point at one line and say "that one". If they have to email you to ask which version they need, the page is not finished.

Page 3 — What it costs

The question: Can I afford this, roughly?

This is the page most small company sites refuse to write, and it is the reason most of them lose the enquiry. If you do not answer it, the visitor assumes you are expensive and goes somewhere that did.

Three acceptable forms, in order of preference:

1. Real prices with the unit stated. Fine for standardised products and fixed-scope services.
2. A starting point. "From X" plus one sentence on what pushes it up.
3. What drives the price. A short list — size, materials, quantity, access, timeline — plus the smallest and largest job you typically take.

What is not acceptable: "contact us for a quote" and nothing else. That is not a price page, it is a form.

Done test: a stranger can place themselves in a band — small, typical, large — without asking you.

Page 4 — About

The question: Who am I dealing with, and can I check them?

What goes on it:

• the legal name of the business, plus any trading name
• the address you actually operate from — a real one
• names and photos of the people a customer will deal with, if that is how you work
• company registration or tax number, where your jurisdiction expects it published
• how long you have been doing this — only if it is a fact you can evidence
• the two or three things you do differently, stated as operations, not adjectives. "We answer email within one working day" is checkable. "We are passionate" is not.

Done test: someone can decide whether you are a real, findable business without leaving the page.

Page 5 — Contact

The question: How do I reach a human, and what happens after I do?

• a phone number that a person answers, with the hours and the timezone
• an email address on your own domain — not a free mailbox
• a form, as an addition, never as the only route
• the physical address, again
• one line on response time, and one line on what to include in the message so the first reply is useful: product, quantity, location, deadline

Done test: send yourself a message through every route on the page and confirm it arrives, and confirm the confirmation screen appears. A contact form that silently fails is the most expensive bug on any small site.

Page 6 — The legal and trust layer

The question: Is this site legitimate, and what do they do with my data?

Privacy policy, terms of sale or service, and — depending on where you are registered — an imprint or legal notice. If you set cookies beyond the strictly necessary, a cookie notice with a real reject option. Every one of these has to be reachable from every page, usually in the footer.

Done test: there is a footer link to each, and each page names the legal entity and a contact route.

The whole thing as a table

What can wait — and the trigger for adding it

Every page below is a real page with real value, and every one of them is a liability if it launches empty. The trigger column is the honest test.

Notice that none of these triggers is "the site looks thin". Thin is a design opinion. Missing the price page is a lost enquiry.

The five-question test for any page

Before you write a seventh page, run it through this. If it fails two or more, do not build it.

1. Does a real visitor arrive with a question this page answers?
2. Could a competitor paste the same text onto their site unchanged? If yes, it is not your page.
3. Is there anything on it that will still be true in a year?
4. Does it end with a next step?
5. Can you maintain it without a project?

Build order

Step 1 — Answer the questions in a text file, not a page builder. Six headings, plain text, no design. If you cannot write the Home sentence in this file, no layout will save it.

Step 2 — Gather assets before you build. Logo in a vector format and a PNG, three to six real photographs, one product photo per variant on a plain background, the price list, the legal entity details. Missing assets are what turn a two-week build into a two-month one.

Step 3 — Draft each page at 200–400 words. Shorter than you think. Long pages are usually a short page with the same idea repeated three ways.

Step 4 — Build in the order above. Home, offer, price, about, contact, legal. Do not start on the design of the blog.

Step 5 — Test before anyone sees it. Open every page on a phone. Submit every form and confirm the email arrives. Click every footer link. Check that every page has exactly one obvious next step and that it is not a dead end.

Traps that cost the most

• The form-only contact page. People who are ready to buy want to call. A form is for people who are not.
• A price page that says "request a quote". It reads as "expensive" to anyone who does not already know you.
• An About page with no names, no address, no registration number. It reads as a hobby.
• A stock photo where a real one would do. A slightly imperfect photo of your actual workshop beats a perfect photo of someone else's.
• Filling the gaps with a blog. Three thin posts do not make a site look established; they make it look abandoned.

Write the six. Ship the six. Add a seventh when a real visitor asks for it, not before.

Here is the whole thing as a process you can run in a week.

Step 1 — Decide the list's single job (15 minutes, before any design)

Write one sentence: "If you give me your email, I will send you ___ , roughly ___ often." If you cannot fill both blanks without hedging, stop here. A form without that sentence collects addresses that go cold in a month.

Two shapes work for a technical site:

• The artifact list. One specific thing — a reference sheet, a checklist, a config template, a benchmark script. You send it immediately, then follow up with related material. High signup rate, small list, and it can stay small.
• The publication list. New long-form posts, sent when they exist. Lower signup rate, better retention, and it survives the day you run out of artifacts.

Pick one. A list that mixes "here is your download" with "monthly newsletter" produces unsubscribes on email two, because the reader signed up for a transaction and got a subscription.

Step 2 — Place the form where intent already exists

Rank placements by what the reader is doing at that moment, not by how much space the layout has.

Rules that hold regardless of layout:

1. One capture point per page. Two forms on the same page split attention and your measurement.
2. Never gate your main search entry point. If a page brings in organic traffic, the content stays open and the form offers the next step.
3. Never put the form above the first code block or the first example. The reader has not received value yet, so the ask is unearned.
4. Name the artifact in the button, not the word "newsletter". "Send me the checklist" beats "Subscribe".

Step 3 — Build the form (and the fields you will regret skipping)

Checklist for the form itself:

• One visible field. Every extra field costs submissions; if you need to segment, do it from behaviour or a hidden field, not from a question.
• Label above the input, always visible. A placeholder that disappears is not a label.
• type="email", autocomplete="email", inputmode="email" so mobile keyboards behave.
• Button text states the outcome.
• One line of fine print: how often, and how to leave.
• A success state that replaces the form and tells the reader what happens next ("check your inbox, the confirm link is there").
• Honeypot field plus a timestamp check instead of a CAPTCHA. CAPTCHAs cost more real signups than they save.
• A hidden source field carrying the page slug. This is the only way you will ever know which placement worked.

A form you can paste into a template and wire to your own handler:

<form action='/subscribe' method='post' class='capture'>
  <label for='email'>Where should I send the checklist?</label>
  <input id='email' name='email' type='email'
         autocomplete='email' inputmode='email' required>

  <input type='hidden' name='source' value='post-template-cache'>

  <!-- honeypot: hidden from people, tempting to bots -->
  <div aria-hidden='true' style='position:absolute;left:-9999px'>
    <label for='website'>Website</label>
    <input id='website' name='website' type='text' tabindex='-1' autocomplete='off'>
  </div>

  <button type='submit'>Send me the checklist</button>
  <p class='fineprint'>One email when there is something worth your time. One-click unsubscribe in every message.</p>
</form>

Server side, reject the submission if the honeypot is filled or if the form was posted in under about two seconds. Store the address, the timestamp, the source value, and a consent flag. Do not store more than you need — an IP address is personal data in several jurisdictions.

Step 4 — The confirm and delivery step

Use double opt-in. Not for legal theatre: it filters typos, blocks list-poisoning, and gives you a record of consent with a date and a source page attached.

Sequence of events on submit:

1. Store the address as pending with source and timestamp.
2. Send the confirmation immediately, in plain text, with one link. No images, no header graphics, no second call to action.
3. On confirm, send the artifact in the same session — do not make the reader wait for a scheduled campaign.
4. Set the sending domain up before you send volume: SPF, DKIM and a DMARC record on the subdomain you send from. Skipping this is the most common reason a working form produces signups nobody ever receives.
5. Keep marketing mail on its own subdomain or stream, separate from password resets and receipts. One complaint spike should not take down transactional mail.

Every message, including the first, carries a working one-click unsubscribe. Make sure the reply-to is a real inbox you read — replies are the highest-value signal you will get.

Step 5 — The follow-up, which is the actual product

Five messages, then a rhythm. Keep each one to a single idea and readable on a phone without scrolling twice.

Rules that keep it from becoming noise:

• One idea per email. Two links maximum, and only if the second is the source.
• Plain text or near-plain. A wall of images and buttons reads as bulk mail to both filters and people.
• Write the subject as the specific content, not as curiosity bait.
• If someone has not clicked anything in about two months, move them to a low-frequency segment rather than deleting them. Deletion is irreversible; a quiet segment is not.
• Turn off the sequence the moment someone replies. A human reply outranks any automation.

Step 6 — Measure per placement, not overall

Group submissions by the hidden source field. The numbers worth watching, in order:

1. Confirm rate — confirmations divided by submissions. If it is low, the promise and the delivered artifact do not match, or the confirm mail is landing in spam.
2. Click rate on the Day 1 email. This is the honest test of whether the list is alive.
3. Reply rate on the Day 14 email. Low volume, high signal.
4. Unsubscribes per send, as a trend, not a single number.

Open rate is unreliable now that clients pre-fetch images. Do not make decisions on it.

Do not judge a placement on a handful of submissions — you cannot tell two placements apart until each has a few hundred, and most personal sites never get there. In that case, pick the placement by intent (end of article, artifact page) and stop optimising.

A one-week checklist

• One sentence written: what they get, how often.
• One artifact exists and is ready to send before the form goes live.
• Form placed at the end of the two best articles, not in the header.
• Single field, visible label, outcome-shaped button text, fine print with frequency and exit.
• Honeypot plus timing check; no CAPTCHA.
• Hidden source field on every instance, different value per placement.
• Double opt-in, consent stored with date and source.
• SPF, DKIM, DMARC on the sending subdomain, tested by sending to yourself at two providers.
• One-click unsubscribe in every message, including the first.
• Emails 1 through 5 written, not just the first one.
• Reply-to is an inbox you actually read.

Short version, before the code:

• No prices, no cart, no stock — spec sheets and a quote form → your own CPT is lighter, the admin screen is yours, and you are not maintaining a shop you do not use.
• You will take payment, need variants, stock counts, tax, or a CSV round-trip → start from products, even if you hide the cart on day one. Moving a custom catalog into products later is a data migration, not a settings change.

1. What actually differs

The row that decides most projects is the first one. Sorting 2,000 products by price through meta_key means a JOIN on wp_postmeta on every archive page. WooCommerce keeps wc_product_meta_lookup (product_id, sku, price, regular_price, sale_price, stock_status, stock_quantity, rating averages) so that sorting and filtering by price or stock never touches postmeta. If price is part of your catalog's browsing experience, that table is the feature.

2. The CPT route

This is one small plugin file. Drop it in wp-content/plugins/catalog-cpt/catalog-cpt.php, or in mu-plugins if you never want it deactivated.

<?php
/**
 * Plugin Name: Catalog CPT
 */

add_action( 'init', function () {
	register_post_type( 'catalog_item', [
		'labels'       => [
			'name'          => 'Catalog items',
			'singular_name' => 'Catalog item',
		],
		'public'       => true,
		'has_archive'  => true,
		'menu_icon'    => 'dashicons-products',
		'supports'     => [ 'title', 'editor', 'thumbnail', 'excerpt', 'custom-fields' ],
		'show_in_rest' => true,
		'rewrite'      => [ 'slug' => 'catalog', 'with_front' => false ],
	] );

	register_taxonomy( 'catalog_family', 'catalog_item', [
		'label'        => 'Families',
		'public'       => true,
		'hierarchical' => true,
		'show_in_rest' => true,
		'rewrite'      => [ 'slug' => 'catalog-family', 'with_front' => false ],
	] );
} );

Rewrite rules need a flush once. Visit Settings → Permalinks and save, or if this is a normal plugin add:

register_activation_hook( __FILE__, function () {
	flush_rewrite_rules();
} );

Now the fields. Register them so REST and the block editor both see them, then add a plain meta box for the classic screen:

add_action( 'init', function () {
	$fields = [
		'_catalog_sku'          => 'string',
		'_catalog_power'        => 'number',
		'_catalog_datasheet_id' => 'integer',
	];

	foreach ( $fields as $key => $type ) {
		register_post_meta( 'catalog_item', $key, [
			'type'              => $type,
			'single'            => true,
			'show_in_rest'      => true,
			'sanitize_callback' => ( 'number' === $type || 'integer' === $type ) ? 'absint' : 'sanitize_text_field',
			'auth_callback'     => function () {
				return current_user_can( 'edit_posts' );
			},
		] );
	}
} );

add_action( 'add_meta_boxes', function () {
	add_meta_box( 'catalog_fields', 'Catalog fields', 'my_catalog_box', 'catalog_item', 'normal', 'high' );
} );

function my_catalog_box( $post ) {
	wp_nonce_field( 'my_catalog_save', 'my_catalog_nonce' );

	$sku   = get_post_meta( $post->ID, '_catalog_sku', true );
	$power = get_post_meta( $post->ID, '_catalog_power', true );

	printf(
		'<p><label>SKU<br /><input type="text" class="widefat" name="catalog_sku" value="%s" /></label></p>',
		esc_attr( $sku )
	);
	printf(
		'<p><label>Power (W)<br /><input type="number" class="widefat" name="catalog_power" value="%s" /></label></p>',
		esc_attr( $power )
	);
}

add_action( 'save_post_catalog_item', function ( $post_id ) {
	if ( defined( 'DOING_AUTOSAVE' ) && DOING_AUTOSAVE ) {
		return;
	}
	if ( ! isset( $_POST['my_catalog_nonce'] ) || ! wp_verify_nonce( $_POST['my_catalog_nonce'], 'my_catalog_save' ) ) {
		return;
	}
	if ( ! current_user_can( 'edit_post', $post_id ) ) {
		return;
	}

	if ( isset( $_POST['catalog_sku'] ) ) {
		update_post_meta( $post_id, '_catalog_sku', sanitize_text_field( wp_unslash( $_POST['catalog_sku'] ) ) );
	}
	if ( isset( $_POST['catalog_power'] ) ) {
		update_post_meta( $post_id, '_catalog_power', absint( $_POST['catalog_power'] ) );
	}
} );

Two things bite people here. First, register_post_meta only exposes the value in the REST response under meta — it does not render an input. Without a meta box or a block-editor field, editors can't set it. Second, the sanitize callback runs on every update_post_meta call too, so do not point it at a function that expects raw $_POST superglobals.

3. The Woo route

The equivalent registration is not a register_post_type call — WooCommerce already registered product. What you register is your spec fields. Keep browsable values in attributes (taxonomies) and the rest in meta:

add_action( 'init', function () {
	register_post_meta( 'product', '_datasheet_id', [
		'type'              => 'integer',
		'single'            => true,
		'show_in_rest'      => true,
		'sanitize_callback' => 'absint',
		'auth_callback'     => function () {
			return current_user_can( 'edit_products' );
		},
	] );
} );

Anything a visitor will filter by should be a global product attribute (Products → Attributes), because that gives you a pa_* taxonomy you can query. Anything only shown on the product page — datasheet ID, dielectric, tolerance band — is fine as post meta.

4. Queries

CPT, sorted by a spec field, filtered by taxonomy:

$query = new WP_Query( [
	'post_type'      => 'catalog_item',
	'posts_per_page' => 24,
	'meta_key'       => '_catalog_power',
	'orderby'        => 'meta_value_num',
	'order'          => 'ASC',
	'tax_query'      => [
		[
			'taxonomy' => 'catalog_family',
			'field'    => 'slug',
			'terms'    => [ 'mosfets' ],
		],
	],
] );

foreach ( $query->posts as $post ) {
	printf(
		'<li>%s — %s W</li>',
		esc_html( get_the_title( $post ) ),
		esc_html( get_post_meta( $post->ID, '_catalog_power', true ) )
	);
}

The gotcha: using meta_key for ordering switches that query to an INNER JOIN on postmeta, so posts without _catalog_power silently disappear. If some items have no value, either backfill them or sort in PHP.

CPT, numeric range filter:

'meta_query' => [
	[
		'key'     => '_catalog_power',
		'value'   => [ 50, 150 ],
		'compare' => 'BETWEEN',
		'type'    => 'NUMERIC',
	],
],

Products, sorted by price:

$products = wc_get_products( [
	'status'  => 'publish',
	'limit'   => 24,
	'orderby' => 'price',
	'order'   => 'ASC',
	'return'  => 'objects',
] );

foreach ( $products as $product ) {
	printf(
		'<li>%s — %s</li>',
		esc_html( $product->get_name() ),
		wp_kses_post( wc_price( $product->get_price() ) )
	);
}

wc_get_products() resolves orderby => 'price' through the lookup table. If you instead run WP_Query on post_type => product with meta_key => '_price', you get the postmeta join again and lose the point of the shop tables.

Products, filtered by category and a global attribute — use WP_Query for the taxonomy work, then wrap the IDs:

$q = new WP_Query( [
	'post_type'      => 'product',
	'posts_per_page' => 24,
	'fields'         => 'ids',
	'tax_query'      => [
		'relation' => 'AND',
		[ 'taxonomy' => 'product_cat', 'field' => 'slug', 'terms' => [ 'mosfets' ] ],
		[ 'taxonomy' => 'pa_package',  'field' => 'slug', 'terms' => [ 'to-247' ] ],
	],
] );

foreach ( $q->posts as $id ) {
	$product = wc_get_product( $id );
	if ( $product ) {
		echo esc_html( $product->get_name() );
	}
}

5. What you gain and give up in each direction

CPT gives you: a clean admin, no cart code loading on catalog pages, total control over the URL structure and the single-item template, and no obligation to keep shop tables consistent.

CPT costs you: variants, stock, tax, coupons, order emails, the CSV round-trip, and the whole woocommerce_* hook ecosystem. Every one of those becomes your code.

Products give you: the data layer, the REST endpoints, the importer, and template overrides that live in yourtheme/woocommerce/ where a theme developer expects them.

Products cost you: more tables, more queries on unrelated pages, a heavier admin, and the constant temptation to install something that changes price output via woocommerce_get_price_html and breaks your spec display.

6. Decision rules

Ask these three questions before you write the first line of registration code:

1. Will money ever change hands through the site? If yes — products.
2. Will a visitor ever need to see "in stock" or pick between variants of the same part? If yes — products.
3. Does your catalog exist to generate a quote request, with prices "on request" and no checkout in sight? If yes — your own CPT is the cheaper build, and staying on it is fine.

If the answer to 1 or 2 is "maybe next year", start on products and hide the cart with a filter until you launch the shop. Converting catalog_item posts into products later means re-creating every item through WC_Product_Simple, remapping taxonomies to product_cat and pa_*, and re-pointing every internal link — a day of work for a small catalog, a project for a large one.

If you are handing the catalog layer to someone else because the spec sheets and the quoting flow are the actual work, that sits under custom development rather than a template install.

One last thing on REST: with show_in_rest => true on the CPT and on the meta, /wp-json/wp/v2/catalog_item gives you both the post and a meta object in one response, which is usually all a front-end needs. The products endpoint requires WooCommerce's key/secret pair, so plan the auth before you promise a client a headless front end.

What each signal can and cannot tell you

The practical rule: treat the UA as a label, the IP range as verification, and frequency and paths as behavior clues.

Step 1: Make sure the log has the fields you need

Use an access log format that includes the request line, status, referrer, and user agent. The common combined format has all of these. If the site sits behind a reverse proxy or CDN, configure the origin to log the original client IP from a trusted header; otherwise every request may look like it came from the proxy.

You do not need to log request bodies or cookies for this. Do not start logging secrets just to identify crawlers.

Step 2: Build a small IP range file

Create crawler-ips.txt. Each line is a crawler name and a CIDR range. Comments start with #.

# Placeholders. Replace with current ranges from each operator's crawler documentation.
GPTBot 203.0.113.0/24
ClaudeBot 198.51.100.0/24

The ranges above are documentation examples, not live crawler ranges. Check the crawler operator's official crawler documentation for the current list. If you cannot find a published range for a token, leave it out of the file and keep that token in the unverified bucket.

Step 3: Run a log audit script

Save this as ai_crawler_audit.py. It uses only the Python standard library.

#!/usr/bin/env python3
import shlex
import sys
from collections import Counter, defaultdict
from datetime import datetime
from ipaddress import ip_address, ip_network
from statistics import median

AI_TOKENS = ('GPTBot', 'ClaudeBot')
MONITOR_HINTS = (
    'uptime', 'monitor', 'health', 'ping', 'probe',
    'curl', 'wget', 'python-requests', 'go-http-client', 'libwww',
)

def load_networks(path):
    networks = defaultdict(list)
    with open(path) as f:
        for line in f:
            line = line.strip()
            if not line or line.startswith('#'):
                continue
            name, cidr = line.split()
            networks[name].append(ip_network(cidr))
    return networks

def ai_token(ua):
    low = ua.lower()
    for token in AI_TOKENS:
        if token.lower() in low:
            return token
    return None

def monitor_like(rec):
    ua = rec['ua'].lower()
    if any(h in ua for h in MONITOR_HINTS):
        return True
    path = rec['path'].split('?', 1)[0]
    if rec['method'] == 'HEAD' and path in ('/', '/health', '/healthz', '/status', '/ping'):
        return True
    return False

def parse_line(line):
    try:
        parts = shlex.split(line, comments=False, posix=True)
    except ValueError:
        return None
    if len(parts) < 10:
        return None
    try:
        ip_obj = ip_address(parts[0])
        ts = parts[3].lstrip('[') + ' ' + parts[4].rstrip(']')
        ts_dt = datetime.strptime(ts, '%d/%b/%Y:%H:%M:%S %z')
        method, path, _ = parts[5].split(' ', 2)
    except ValueError:
        return None
    return {
        'ip': parts[0],
        'ip_obj': ip_obj,
        'ts': ts_dt,
        'method': method,
        'path': path,
        'status': parts[6],
        'ua': parts[9],
    }

def group_for(rec, networks):
    token = ai_token(rec['ua'])
    if token:
        verified = any(rec['ip_obj'] in net for net in networks.get(token, []))
        return token if verified else f'{token} (unverified UA)'
    if monitor_like(rec):
        return 'monitor candidate'
    return 'other'

def summarize(records):
    by_ip = defaultdict(list)
    for r in records:
        by_ip[r['ip']].append(r['ts'])
    gaps = []
    for times in by_ip.values():
        times.sort()
        gaps.extend((b - a).total_seconds() for a, b in zip(times, times[1:]))
    paths = Counter(r['path'].split('?', 1)[0] for r in records)
    minutes = Counter(r['ts'].strftime('%Y-%m-%d %H:%M') for r in records)
    return {
        'requests': len(records),
        'unique_ips': len(by_ip),
        'unique_paths': len(paths),
        'methods': dict(Counter(r['method'] for r in records)),
        'status_classes': dict(Counter(r['status'][0] + 'xx' for r in records)),
        'top_paths': paths.most_common(5),
        'peak_requests_per_minute': max(minutes.values(), default=0),
        'median_gap_same_ip_s': round(median(gaps), 1) if gaps else None,
        'max_requests_one_ip': max((len(v) for v in by_ip.values()), default=0),
    }

def main():
    if len(sys.argv) != 3:
        print('usage: python3 ai_crawler_audit.py access.log crawler-ips.txt')
        return 1
    networks = load_networks(sys.argv[2])
    groups = defaultdict(list)
    with open(sys.argv[1], errors='replace') as f:
        for line in f:
            rec = parse_line(line.rstrip())
            if rec:
                groups[group_for(rec, networks)].append(rec)
    for name, records in sorted(groups.items(), key=lambda item: -len(item[1])):
        print(f'## {name}')
        for key, value in summarize(records).items():
            print(f'{key}: {value}')
        print()
    return 0

if __name__ == '__main__':
    raise SystemExit(main())

Run it against a combined access log:

python3 ai_crawler_audit.py /var/log/nginx/access.log crawler-ips.txt

If your log format is not combined, adjust the field positions in parse_line. Keep the same output fields so the rest of the script still works.

Step 4: Read the groups, not just the totals

The script puts each request into one of these groups:

• GPTBot or ClaudeBot: the UA contains the token and the source IP matches the range file.
• GPTBot (unverified UA) or ClaudeBot (unverified UA): the UA claims the token, but the IP is not in the range file. This is either a stale range file, a proxy, or a spoofed UA. Investigate before counting it as the real crawler.
• monitor candidate: the UA or method looks like uptime or health checking. This is a heuristic, not proof.
• other: everything else.

For each group, look at:

• unique_ips and max_requests_one_ip
• unique_paths and top_paths
• peak_requests_per_minute
• median_gap_same_ip_s
• methods and status_classes

A broad crawler usually has many unique paths and may use many IPs. A simple monitor usually has a small path set, a small IP set, and a regular interval. Use those as clues, then confirm with the UA and IP range.

Step 5 (optional route): skip log parsing and read the hits in WordPress

Log analysis is the route that gives you the most control and the most work. If the site is on WordPress and you do not want to keep a range file current, the free AumCrawl plugin is one option that sits alongside this workflow rather than replacing it: it records which AI crawlers are requesting the site and lets you allow or block each crawler separately. That is useful for the "allow or deny per crawler" decision, which is a policy question rather than a parsing question. It does not give you the same freedom as raw logs when you need to audit a suspicious UA or a proxy setup, so keep the log script for anything you have to prove.

Plugin page: https://aumcreate.com/plugins/aumcrawl?utm_source=hashnode&utm_medium=content&utm_campaign=hashnode-ai-crawler-log-ua-frequency

Either way, decide the policy after you know the facts: which tokens arrive, which of those verify against the operator ranges, and which paths they touch.

How to separate the three cases

GPTBot vs ClaudeBot

Start with the token: GPTBot and ClaudeBot are different strings. Then verify the source IP against the matching range file. Do not use frequency to decide which one it is. A slow crawl from one IP and a fast crawl from many IPs are both possible; the operator and the site configuration change the rate.

If the token is GPTBot but the IP is not in the GPTBot ranges, do not relabel it as ClaudeBot or as a monitor. Keep it unverified. It may be a proxy, a stale range list, or a forged UA.

Crawler vs ordinary monitoring

Look for the combination:

• A monitor usually repeats a small set of paths such as /, /health, /status, /ping, or /robots.txt.
• A monitor often uses HEAD or a simple GET, and may send no cookies or referrer.
• A monitor often comes from one or a few IPs at a fixed interval.
• A broad crawler usually requests many different URLs and may spread across many IPs.
• A crawler may also request only HTML and skip assets, so asset requests are not a reliable test.

No single row is enough. For example, a GET / from a generic HTTP client can be a monitor, a search engine, or a person with a script. The UA plus the IP range plus the path pattern is the useful combination.

Frequency checks that are useful

peak_requests_per_minute tells you whether a group arrives in bursts. median_gap_same_ip_s tells you the typical time between requests from the same IP. Keep them separate:

• Low unique paths plus high peak plus few IPs points to polling or health checks.
• High unique paths plus many IPs points to a broad crawl.
• High unique paths plus one IP can be a small crawler, a mirror, or a scraper.
• Low unique paths plus many IPs can be a distributed monitor or a proxy pool.

Do not turn these into a fixed rule. Logs from different sites will produce different shapes. The point is to form a hypothesis and verify it with the UA and IP range.

Common mistakes

• Trusting the UA alone. It is a claim, not proof.
• Treating a high request rate as proof of GPTBot or ClaudeBot. Frequency is behavior, not identity.
• Forgetting the proxy. If the origin sees only the proxy IP, IP verification will fail.
• Using a stale range file. Refresh it on a schedule.
• Calling every GET / a monitor. Many real users and crawlers request the home page.
• Counting an unverified UA as a confirmed crawler in reports.

Checklist

• The access log includes the user agent and the request path.
• The origin logs the real client IP, not only a proxy IP.
• crawler-ips.txt uses current operator ranges, not the placeholders.
• The audit script runs without errors on a sample log.
• Unverified UA groups are reviewed before they are counted.
• Frequency and path breadth are used as clues, not as identity.
• The range file is refreshed on a schedule.