This has happened to me a couple of times now, so I thought I’d document it.
TrueCommand has indicated that replication is failing on my main server for a particular dataset.
Now, I know I’ve been messing around quite a bit with WordPress, rebuilding jails and associated datasets. The UDance WordPress site has been problematical as I haven’t given it much attention as I’ve been focused on the blog site. This has given hackers the opportunity to mess around with it. So, I’ve rebuilt its jail and datasets a couple of times now, often across the midnight boundary when replication runs.
Having a look at the replication source, I can see the error.
Clicking on the error button and downloading the log file, the contents are:
[2020/09/30 11:17:37] INFO [Thread-244] [zettarepl.paramiko.replication_task__task_36] Connected (version 2.0, client OpenSSH_8.0-hpn14v15)
[2020/09/30 11:17:37] INFO [Thread-244] [zettarepl.paramiko.replication_task__task_36] Authentication (publickey) successful!
[2020/09/30 11:17:52] INFO [replication_task__task_36] [zettarepl.replication.run] No snapshots to send for replication task 'task_36' on dataset 'tank/apps'
[2020/09/30 11:17:52] INFO [replication_task__task_36] [zettarepl.replication.run] No snapshots to send for replication task 'task_36' on dataset 'tank/apps/caddy'
[2020/09/30 11:17:52] INFO [replication_task__task_36] [zettarepl.replication.run] No snapshots to send for replication task 'task_36' on dataset 'tank/apps/caddyv1'
[2020/09/30 11:17:52] INFO [replication_task__task_36] [zettarepl.replication.run] No snapshots to send for replication task 'task_36' on dataset 'tank/apps/dnsmasq'
[2020/09/30 11:17:52] INFO [replication_task__task_36] [zettarepl.replication.run] No snapshots to send for replication task 'task_36' on dataset 'tank/apps/nextcloud'
[2020/09/30 11:17:52] INFO [replication_task__task_36] [zettarepl.replication.run] No snapshots to send for replication task 'task_36' on dataset 'tank/apps/nextcloud/config'
[2020/09/30 11:17:52] INFO [replication_task__task_36] [zettarepl.replication.run] No snapshots to send for replication task 'task_36' on dataset 'tank/apps/nextcloud/db'
[2020/09/30 11:17:52] INFO [replication_task__task_36] [zettarepl.replication.run] No snapshots to send for replication task 'task_36' on dataset 'tank/apps/nextcloud/files'
[2020/09/30 11:17:52] INFO [replication_task__task_36] [zettarepl.replication.run] No snapshots to send for replication task 'task_36' on dataset 'tank/apps/nextcloud/themes'
[2020/09/30 11:17:52] INFO [replication_task__task_36] [zettarepl.replication.run] No snapshots to send for replication task 'task_36' on dataset 'tank/apps/nextcloud2'
[2020/09/30 11:17:52] INFO [replication_task__task_36] [zettarepl.replication.run] No snapshots to send for replication task 'task_36' on dataset 'tank/apps/nextcloud2/config'
[2020/09/30 11:17:52] INFO [replication_task__task_36] [zettarepl.replication.run] No snapshots to send for replication task 'task_36' on dataset 'tank/apps/nextcloud2/db'
[2020/09/30 11:17:52] INFO [replication_task__task_36] [zettarepl.replication.run] No snapshots to send for replication task 'task_36' on dataset 'tank/apps/nextcloud2/files'
[2020/09/30 11:17:52] INFO [replication_task__task_36] [zettarepl.replication.run] No snapshots to send for replication task 'task_36' on dataset 'tank/apps/nextcloud2/themes'
[2020/09/30 11:17:52] INFO [replication_task__task_36] [zettarepl.replication.run] No snapshots to send for replication task 'task_36' on dataset 'tank/apps/plex'
[2020/09/30 11:17:52] INFO [replication_task__task_36] [zettarepl.replication.run] No snapshots to send for replication task 'task_36' on dataset 'tank/apps/rslsync'
[2020/09/30 11:17:52] INFO [replication_task__task_36] [zettarepl.replication.run] No snapshots to send for replication task 'task_36' on dataset 'tank/apps/rslsync/config'
[2020/09/30 11:17:52] INFO [replication_task__task_36] [zettarepl.replication.run] No snapshots to send for replication task 'task_36' on dataset 'tank/apps/rslsync/data'
[2020/09/30 11:17:52] INFO [replication_task__task_36] [zettarepl.replication.run] No snapshots to send for replication task 'task_36' on dataset 'tank/apps/tautulli'
[2020/09/30 11:17:52] INFO [replication_task__task_36] [zettarepl.replication.run] No snapshots to send for replication task 'task_36' on dataset 'tank/apps/transmission'
[2020/09/30 11:17:52] INFO [replication_task__task_36] [zettarepl.replication.run] No snapshots to send for replication task 'task_36' on dataset 'tank/apps/transmission/config'
[2020/09/30 11:17:52] INFO [replication_task__task_36] [zettarepl.replication.run] No snapshots to send for replication task 'task_36' on dataset 'tank/apps/transmission/downloads'
[2020/09/30 11:17:52] INFO [replication_task__task_36] [zettarepl.replication.run] No snapshots to send for replication task 'task_36' on dataset 'tank/apps/wordpress'
[2020/09/30 11:17:52] INFO [replication_task__task_36] [zettarepl.replication.run] No snapshots to send for replication task 'task_36' on dataset 'tank/apps/wordpress/barry'
[2020/09/30 11:17:52] INFO [replication_task__task_36] [zettarepl.replication.run] No snapshots to send for replication task 'task_36' on dataset 'tank/apps/wordpress/barry/db'
[2020/09/30 11:17:52] INFO [replication_task__task_36] [zettarepl.replication.run] No snapshots to send for replication task 'task_36' on dataset 'tank/apps/wordpress/barry/files'
[2020/09/30 11:17:52] INFO [replication_task__task_36] [zettarepl.replication.run] No snapshots to send for replication task 'task_36' on dataset 'tank/apps/wordpress/blog'
[2020/09/30 11:17:52] INFO [replication_task__task_36] [zettarepl.replication.run] No snapshots to send for replication task 'task_36' on dataset 'tank/apps/wordpress/blog/db'
[2020/09/30 11:17:52] INFO [replication_task__task_36] [zettarepl.replication.run] No snapshots to send for replication task 'task_36' on dataset 'tank/apps/wordpress/blog/files'
[2020/09/30 11:17:52] ERROR [replication_task__task_36] [zettarepl.replication.run] For task 'task_36' non-recoverable replication error NoIncrementalBaseReplicationError("No incremental base on dataset 'tank/apps/wordpress/udance' and replication from scratch is not allowed")Replication fo the udance child dataset appears to be the issue (last line in the log).
Here’s a view of snapshots on the replication source…
… and on the replication target.
Next step is to delete the udance sub-datasets on the replication target.
This time, running the replication task on the replication source…
… produces a successful outcome.
Checking the snapshots on the replication target, we see that the source snapshots have been transferred..
So, if this issue arises, as it has on a couple of occasions, the key is to delete the offending target dataset and rerun replication. This issue would not happen under normal operating conditions. It’s only occurred because of a lot of startup and pulldown activity on WordPress jails.
Comments