danh
/
major-projects-grabber
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
Download documents from the NSW DPE Major Projects website.
You cannot select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
11 Commits
1 Branch
0 Tags
336 KiB
Python 100%
 
master
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from '1e27a4d77f'
${ noResults }
Go to file
Dan Howe 1e27a4d77f Add support for folder paths longer than 255 characters (Windows) 6 years ago
major_projects_grabber Add support for folder paths longer than 255 characters (Windows) 6 years ago
README.md Add readme 6 years ago
setup.py Add tdqm dependency 6 years ago

README.md

major_projects_grabber

major_projects_grabber download documents from the NSW DPE Major Projects website. http://majorprojects.planning.nsw.gov.au

Installation

http://git.wrl.unsw.edu.au:3000/danh/major-projects-grabber.git
pip install -e major-projects-grabber

Usage

Single modification

The URL of each modification page on the DPE Major Projects website ends with a job id, e.g.:

majorprojects.planning.nsw.gov.au/index.pl?action=view_job&job_id=1746

This job id can be used to download all documents associated with a particular modification:

major_projects_grabber -i 1746

List of search results

The DPE portal's search function takes you to a page of search results with a URL like this:

http://majorprojects.planning.nsw.gov.au/index.pl?action=search&page_id=&search=&authority_id=425&search_site_type_id=&reference_table=&status_id=&decider=&from_date=&to_date=&x=44&y=10

This URL can be used to download all documents for each modifications in the search results page:

major_projects_grabber -u "http://majorprojects.planning.nsw.gov.au/index.pl?action=search&authority_id=547"

Output directory

The output directory to save the downloaded documents can be specified with the -o option, e.g.:

major_projects_grabber -i 1746 -o "DPE documents"

If no output directory is specified, the current directory is used.