# CoastSnap Automation ## About this repo This repo aims to automate the workflow to generate timelapses and do shoreline mapping with CoastSnap (Spotteron) images. Check out `workflow.pptx` for an informative schematic of the intended workflow. Batch downloading code is an extension on the Leaman CoastSnap Toolbox (https://git.wrl.unsw.edu.au/chrisl/coastsnap.git) This repo can: * DOWNLOAD images stored on the Spotteron app service * REGISTER the downloaded images in photoshop * TAG registered images with the image date, time, tide and contributor  ## Getting started ### Prerequisites This package has only been tested on a Windows system. Using the [Anaconda distribution](https://www.anaconda.com/products/individual) of Python is recommended for easiest installation of the toolbox requirements. ### Installation 1. Install Git (if you don't already have it) for windows from the following link https://git-scm.com/download/win 2. Clone this repo onto your PC: `git clone https://git.wrl.unsw.edu.au/jonoc/CoastsnapAuto.git' 3. Change your working directory into the repo: `cd CoastsnapAuto` 4. Update or create a new conda environment called "coastsnap" with the required dependencies. This is (almost) the same environment as for the Leaman CoastSnap Toolbox, so you may already have it. If you're not sure, run: `conda info --envs` and look for "coastsnap". If you already have the environment, run: `conda env update --name coastsnap --file environment.yml` to update it. Otherwise, create the environment by runnning: `conda env create -f environment.yml`. 5. Check if you can activate the conda environment with `activate coastsnap`. If not, you may need to update your environment `Path` variable to include paths to the Anaconda install, like the following: ```sh C:\Users\z5079346\Applications\Anaconda C:\Users\z5079346\Applications\Anaconda\Scripts C:\Users\z5079346\Applications\Anaconda\Library\bin ``` If `activate coastsnap` is still not working, you will have to edit each of the .bat files and substitute the first line `call activate coastsnap` with the direct path to the `activate.bat` file, such as "C:\Users\z5079346\Applications\Anaconda\Scripts\activate.bat". Substitute the third line `call deactivate` with the path to `deactivate.bat` ### Demo (optional) The following is a demo to make sure all of the code is working and for the user to get a sense of the intended workflow. It'll take approx. 10 mins (assuming everything works). The code will run on a demo CoastSnap directory (`CoastSnap_DEMO`) with 3 sites. 1. Open up `coastsnap_sites.csv`. Change the path name in cell E2, to the path of the CoastsnapAuto/CoastSnap_DEMO directory. 2. Open the CoastsnapAuto/coastsnap directory 3. Run `1_batch_spotteron_download.bat` [30s] - This downloads 5 images each for 3 sites and stores them in CoastSnap_DEMO/Images/Site_Name/Processed) 4. Run `2_batch_photoshop_registration.bat` [2 mins] - Note: If this doesn't work, edit the .bat file and make sure the path to your photoshop is correct. - This registers the 15 images in photoshop and stores them in CoastSnap_DEMO/Images/Site_Name/Photoshop 5. Run `3_batch_tag_registered.bat` [3 mins] - This tags the photoshop registered images and stores them in CoastSnap_DEMO/Images/Site_Name/Registered. Note: Loading .mat tide data in Python takes a while. 6. Delete any poorly registered/tagged images in CoastSnap_DEMO/Images/Site_Name/Registered 7. Run `4_generate_images_snapshot.bat` - This generates a snapshot of the CoastSnap images directory at `images_snapshot.csv` ### Update file paths To run this code on the shared oneDrive CoastSnap directory: 1. Delete or rename `coastsnap_sites.csv` 2. Rename `coastsnap_sites_ALL.csv` to `coastsnap_sites.csv` 3. Open `coastsnap_sites.csv` and change the path name in cell E2, to the oneDrive CoastSnap directory. NOTE: `coastsnap_sites.csv` is called by all of the scripts in this repo. DON'T CHANGE THE STRUCTURE OF THE CSV, as it'll cause the registration.jsx scripts to die. Feel free to add more sites (rows) and change the limits though. ### Repo updates If you want this repo's latest code, do the following: 1. Open your terminal and make sure you're in the directory you cloned the coastsnap repo into. 2. Ensure you have the coastsnap conda environment active: `activate coastsnap` 3. Pull the latest changes: `git pull` 4. Update the environment: `conda env update --name coastsnap --file environment.yml` ## Usage ### 1. Download images from Spotteron Run `1_batch_spotteron_download.bat` Script Logic: For every site in `coastsnap_sites.csv`, download the latest Spotteron images and save to Processed. * Starts downloading from the most recent image and stops downloading for the site when an image has already been downloaded, to avoid overwriting images. ### 2. Photoshop image registration #### Batch register Run `2_batch_photoshop_registration.bat` Script Logic: For every site in `coastsnap_sites.csv`, iterate through the sites' Processed images (Images/Site/Processed). For any image that hasn't already been photoshop registered (in Images/Site/Photoshop), register it. * To get rid of the popup error "You are about to run the script containing...." when running this script, which you'll want to do if running this as a scheduled task: 1. Create a text file 2. Add this to the file ```sh # Force VM Buffering VMForceBuffering 1 WarnRunningScripts 0 ``` 3. Call it PSUserConfig.txt and save as C:\Users\[Username]\AppData\Roaming\Adobe\Adobe Photoshop [Version]\Adobe Photoshop [Version] Settings\PSUserConfig.txt (Replacing Username and Version. Note, if you can't find the folder 'AppData', you may need to check 'show hidden folders') * This script is based on the manual photoshop registration process, but additionally resizes images to the same width or height of the Target image prior to registration: https://unsw.sharepoint.com/:w:/r/sites/Coastsnap/_layouts/15/doc2.aspx?action=edit&sourcedoc=%7B49c9d377-f450-40bf-93e4-6ff0f289e889%7D&wdOrigin=TEAMS-ELECTRON.teams.chiclet&wdExp=TEAMS-CONTROL * Output images are all 1280px wide and will have the same aspect ratio as the sites' Target.jpg image * Batch Size: Images are registered with the target and seed images, in batches of 15. This is set on line 13 of `photoshop_registration_all_sites.jsx` and can be changed. Images may be registered in smaller batch sizes if there are less than 15 images in the Site/Processed/Year folder. ### 3. Tag Images #### Batch tag images Run `3_batch_tag_registered.bat` Script Logic: For every site in oneDrive CoastSnap directory, iterate through the sites' registered images (Images/Site/Photoshop). Tag the images with their Date, Time, Contributor and Tide (if tide data for site exists), suffix the filename with "_registered" and save at Images/Site/Registered. * Starts tagging from the most recent image and stops for the site when an image has already been tagged. This way, the user can manually remove bad registered/tagged images, and they will not be automatically replaced. * Retrieves tide data for the site from the .mat file specified in Database/CoastSnapDB.xlsx #### Manual tagging If you want a bit more control over your image tagging, you'll have to do it from the command line. `cd` to this repo and activate the coastsnap conda environment with `activate coastsnap`. Now `cd` to the `CoastsnapAuto/coastsnap` directory. From here, your options are: 1. Tag images for a specific site `python tag_registered.py --site [SITE_NAME]` 2. Tag images that have been reregistered (i.e. Deleted from the Photoshop directory and registered again) `python tag_registered.py --tag_reregistered` 3. Tag images that have been deleted from the Registered directory `python tag_registered.py --tag_registered_deleted` 4. Overwrite tagged images in the Registered directory `python tag_registered.py --overwrite` You can also combine these options, for instance if you want to tag reregistered images for blacksmiths you'd run: `python tag_registered.py --site blacksmiths --tag_reregistered` #### ### 4. Images Snapshot Run `generate_images_snapshot.bat` Generates `4_images_snapshot.csv` which contains information about the CoastSnap oneDrive Images directory. Columns include: site | # processed | # photoshop | # registered | stability | most recently deleted * stability (%) = # registered / # photoshop. This formula is based on the assumption that someone will manually remove poorly registered images in `Images/Registered`. Thus stability represents the percentage of images that had good registration. * most recently deleted. This is the image date of the most recently deleted in `Images/Registered`. Could be useful to know for manual refining, so the user doesn't have to check every image each time. ## Future Improvements ### Image metadata Currently (22/6/22) it would appear that images downloaded from Spotteron do not retain the images' metadata. This is based on looking in windows file explorer image->properties, as well as using the exif python package. Note: The metadata presented in file explorer is IPTC data. There is a python package to interact with this data, but I had issues with it.