Differences

This shows you the differences between two versions of the page.

--- guides:plotting:plotmanlinux [2021/08/22 20:09] – created vleesbanaan
+++ guides:plotting:plotmanlinux [2021/08/26 19:24] (current) – spacefarmer01
@@ Line 1: / Line 1: @@
 ===== Automated plotting on Linux with Plotman =====
+==== Reference Sites ====
+  - Install on plotting machine
+    * https://chiadecentral.com/plotman-automate-and-manage-chia-plotting/
+  - Configure archiver to move plots to external or remote drives
+    * https://github.com/ericaltendorf/plotman/wiki/Archiving
+==== Installing Plotman ====
+  - Ensure that you have at least Python 3.7 installed and Pip installed
+    * For help go to [[https://packaging.python.org/tutorials/installing-packages/]]
+  - Activate your Chia environment
+    * ''. ./activate''
+  - Install Plotman
+    * ''pip install –force-reinstall git+https://github.com/ericaltendorf/plotman@main''
+  - Verify that Plotman is working with
+    * ''plotman status''
+==== Configure Plotting and Destination Drives ====
+  - Create a default plotman.yaml config file
+    * ''plotman config generate''
+    * This will be located in ~/.config/plotman/plotman.yaml
+  - Plotman is designed to maximize the number of plots running in parallel
+    * You need to tell it which drives you use for
+      * plotting (primary and secondary)
+      * destination
+      * (optionally) archiving
+    * And how to stagger jobs and the amount of resources to use
+    * An example config for plotting with two SSD's, a second plotting temp drive, and a single destination drive
+      * <code>directories:
+   # normally NVMe SSD or SAS
+   tmp:
+      - /data/s970evop_1
+      - /data/mp510_1
+   # only worthwhile if NVMe or SATA SSD
+   tmp2: /data/crp1_1/tmp
+   # normally spinning rust
+   dst:
+      - /data/wdredp_1</code>
+==== Configure Job Staggering ====
+  * An example config that allows 6 parallel jobs in total, with no more than 3 jobs per primary plotting drive, but only 2 jobs per drive in phase 1
+    * <code>scheduling:
+   tmpdir_stagger_phase_major: 2
+   tmpdir_stagger_phase_minor: 1
+   tmpdir_stagger_phase_limit: 2   # no more than 2 jobs per drive for above phase thresholds
+   tmpdir_max_jobs: 3
+   global_max_jobs: 6</code>
+==== Configure Plotman Plotting for Pools ====
+  * To allow creation of plots for a pool, add the pool contract address to the config file
+    * <code>plotting:
+   pool_contract_address: xch..............</code>
+    * Find your pool contract address with the command
+      * ''chia plotnft show''
+==== Configure Chia Plot Parameters====
+  * Example for rig with 12 hyperthreads, 32 GB RAM, and max of 6 parallel jobs
+    * <code>plotting:
+type: chia
+chia:
+  k: 32                # k-size of plot, leave at 32 most of the time
+  e: False             # Use -e plotting option
+  n_threads: 4         # Threads per job: Can overcommit providing no more than 4 jobs allowed in phase 1
+  n_buckets: 128       # Number of buckets to split data into
+  job_buffer: 5120     # Per job memory: Increase above default 3390 if RAM to spare for marginal improvement</code>
+==== Configure Archiving ====
+  * Plotman allows optional archiving to either local or remote drives
+    * Although rsyncd is recommended for remote drives, it is also possible to use remote NFS drives and Plotman will think that they are local drives
+    * An example config for archiving with throttled bandwidth to a harvester running on a Raspberry Pi with NFS drives is
+      * <code>archiving:
+  target: rsync_rsync_throttle
+  env:
+    site_root: /nfs/chia
+  target_definitions:
+    rsync_rsync_throttle:
+      env:
+        command: rsync
+        # adjust bandwidth limit in bytes/second to avoid overloading device hosting archive drives
+        options: --bwlimit=35000 --preallocate --remove-source-files --skip-compress plot --whole-file
+        site_root: null
+      disk_space_script: |
+        #!/bin/bash
+        set -evx
+        df -BK | grep " ${site_root}/" | awk '{ gsub(/K$/,"",$4); printf "%s:%.0f\n", $6, $4*1024 }'
+      transfer_script: |
+        #!/bin/bash
+        set -evx
+        "${command}" ${options} "${source}" "${destination}"
+      transfer_process_name: "{command}"
+      transfer_process_argument_prefix: "{site_root}"</code>
+==== Running Plotman ====
+  * From the CLI the three commands you will use the most are
+    * ''plotman status''
+      * Gives an overview of all the jobs running
+    * ''plotman interactive''
+      * Interactive screen showing jobs and archiving progress
+      * Works well with headless systems, i.e. SSH friendly
+      * Stop and start plotting and archiving using the <p> and <a> keys
+      * Quit the interactive screen by pressing <q>
+    * ''plotman analyze''
+      * Allows you to see how many jobs you have created over period
+      * For example:
+      * ''plotman analyze ~/.chia/mainnet/log/*2021-08-09*''
+      * <code>+-------+----+-------------+---------------+--------------+---------------+---------------+--------------+
+| Slice | n  |   %usort    |    phase 1    |   phase 2    |    phase 3    |    phase 4    |  total time  |
++=======+====+=============+===============+==============+===============+===============+==============+
+| x     | 18 | μ=100.0 σ=0 | μ=12.6K σ=659 | μ=5.4K σ=307 | μ=11.6K σ=817 | μ=854.6 σ=132 | μ=30.5K σ=1K |
++-------+----+-------------+---------------+--------------+---------------+---------------+--------------+</code>