This has happened to me a couple of times now, so I thought I’d document it.
TrueCommand has indicated that replication is failing on my main server for a particular dataset.

Now, I know I’ve been messing around quite a bit with WordPress, rebuilding jails and associated datasets. The UDance WordPress site has been problematical as I haven’t given it much attention as I’ve been focused on the blog site. This has given hackers the opportunity to mess around with it. So, I’ve rebuilt its jail and datasets a couple of times now, often across the midnight boundary when replication runs.
Having a look at the replication source, I can see the error.

Clicking on the error button and downloading the log file, the contents are:
[2020/09/30 11:17:37] INFO [Thread-244] [zettarepl.paramiko.replication_task__task_36] Connected (version 2.0, client OpenSSH_8.0-hpn14v15) [2020/09/30 11:17:37] INFO [Thread-244] [zettarepl.paramiko.replication_task__task_36] Authentication (publickey) successful! [2020/09/30 11:17:52] INFO [replication_task__task_36] [zettarepl.replication.run] No snapshots to send for replication task 'task_36' on dataset 'tank/apps' [2020/09/30 11:17:52] INFO [replication_task__task_36] [zettarepl.replication.run] No snapshots to send for replication task 'task_36' on dataset 'tank/apps/caddy' [2020/09/30 11:17:52] INFO [replication_task__task_36] [zettarepl.replication.run] No snapshots to send for replication task 'task_36' on dataset 'tank/apps/caddyv1' [2020/09/30 11:17:52] INFO [replication_task__task_36] [zettarepl.replication.run] No snapshots to send for replication task 'task_36' on dataset 'tank/apps/dnsmasq' [2020/09/30 11:17:52] INFO [replication_task__task_36] [zettarepl.replication.run] No snapshots to send for replication task 'task_36' on dataset 'tank/apps/nextcloud' [2020/09/30 11:17:52] INFO [replication_task__task_36] [zettarepl.replication.run] No snapshots to send for replication task 'task_36' on dataset 'tank/apps/nextcloud/config' [2020/09/30 11:17:52] INFO [replication_task__task_36] [zettarepl.replication.run] No snapshots to send for replication task 'task_36' on dataset 'tank/apps/nextcloud/db' [2020/09/30 11:17:52] INFO [replication_task__task_36] [zettarepl.replication.run] No snapshots to send for replication task 'task_36' on dataset 'tank/apps/nextcloud/files' [2020/09/30 11:17:52] INFO [replication_task__task_36] [zettarepl.replication.run] No snapshots to send for replication task 'task_36' on dataset 'tank/apps/nextcloud/themes' [2020/09/30 11:17:52] INFO [replication_task__task_36] [zettarepl.replication.run] No snapshots to send for replication task 'task_36' on dataset 'tank/apps/nextcloud2' [2020/09/30 11:17:52] INFO [replication_task__task_36] [zettarepl.replication.run] No snapshots to send for replication task 'task_36' on dataset 'tank/apps/nextcloud2/config' [2020/09/30 11:17:52] INFO [replication_task__task_36] [zettarepl.replication.run] No snapshots to send for replication task 'task_36' on dataset 'tank/apps/nextcloud2/db' [2020/09/30 11:17:52] INFO [replication_task__task_36] [zettarepl.replication.run] No snapshots to send for replication task 'task_36' on dataset 'tank/apps/nextcloud2/files' [2020/09/30 11:17:52] INFO [replication_task__task_36] [zettarepl.replication.run] No snapshots to send for replication task 'task_36' on dataset 'tank/apps/nextcloud2/themes' [2020/09/30 11:17:52] INFO [replication_task__task_36] [zettarepl.replication.run] No snapshots to send for replication task 'task_36' on dataset 'tank/apps/plex' [2020/09/30 11:17:52] INFO [replication_task__task_36] [zettarepl.replication.run] No snapshots to send for replication task 'task_36' on dataset 'tank/apps/rslsync' [2020/09/30 11:17:52] INFO [replication_task__task_36] [zettarepl.replication.run] No snapshots to send for replication task 'task_36' on dataset 'tank/apps/rslsync/config' [2020/09/30 11:17:52] INFO [replication_task__task_36] [zettarepl.replication.run] No snapshots to send for replication task 'task_36' on dataset 'tank/apps/rslsync/data' [2020/09/30 11:17:52] INFO [replication_task__task_36] [zettarepl.replication.run] No snapshots to send for replication task 'task_36' on dataset 'tank/apps/tautulli' [2020/09/30 11:17:52] INFO [replication_task__task_36] [zettarepl.replication.run] No snapshots to send for replication task 'task_36' on dataset 'tank/apps/transmission' [2020/09/30 11:17:52] INFO [replication_task__task_36] [zettarepl.replication.run] No snapshots to send for replication task 'task_36' on dataset 'tank/apps/transmission/config' [2020/09/30 11:17:52] INFO [replication_task__task_36] [zettarepl.replication.run] No snapshots to send for replication task 'task_36' on dataset 'tank/apps/transmission/downloads' [2020/09/30 11:17:52] INFO [replication_task__task_36] [zettarepl.replication.run] No snapshots to send for replication task 'task_36' on dataset 'tank/apps/wordpress' [2020/09/30 11:17:52] INFO [replication_task__task_36] [zettarepl.replication.run] No snapshots to send for replication task 'task_36' on dataset 'tank/apps/wordpress/barry' [2020/09/30 11:17:52] INFO [replication_task__task_36] [zettarepl.replication.run] No snapshots to send for replication task 'task_36' on dataset 'tank/apps/wordpress/barry/db' [2020/09/30 11:17:52] INFO [replication_task__task_36] [zettarepl.replication.run] No snapshots to send for replication task 'task_36' on dataset 'tank/apps/wordpress/barry/files' [2020/09/30 11:17:52] INFO [replication_task__task_36] [zettarepl.replication.run] No snapshots to send for replication task 'task_36' on dataset 'tank/apps/wordpress/blog' [2020/09/30 11:17:52] INFO [replication_task__task_36] [zettarepl.replication.run] No snapshots to send for replication task 'task_36' on dataset 'tank/apps/wordpress/blog/db' [2020/09/30 11:17:52] INFO [replication_task__task_36] [zettarepl.replication.run] No snapshots to send for replication task 'task_36' on dataset 'tank/apps/wordpress/blog/files' [2020/09/30 11:17:52] ERROR [replication_task__task_36] [zettarepl.replication.run] For task 'task_36' non-recoverable replication error NoIncrementalBaseReplicationError("No incremental base on dataset 'tank/apps/wordpress/udance' and replication from scratch is not allowed")
Replication fo the udance
child dataset appears to be the issue (last line in the log).
Here’s a view of snapshots on the replication source…

… and on the replication target.

Next step is to delete the udance
sub-datasets on the replication target.

This time, running the replication task on the replication source…

… produces a successful outcome.

Checking the snapshots on the replication target, we see that the source snapshots have been transferred..

So, if this issue arises, as it has on a couple of occasions, the key is to delete the offending target dataset and rerun replication. This issue would not happen under normal operating conditions. It’s only occurred because of a lot of startup and pulldown activity on WordPress jails.
Comments