You cannot select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
|
|
|
|
# major-projects-grabber #
|
|
|
|
|
|
|
|
|
|
**major_projects_grabber** downloads documents from the NSW DPE Major Projects website.
|
|
|
|
|
http://majorprojects.planning.nsw.gov.au
|
|
|
|
|
|
|
|
|
|
## Installation ##
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
|
git clone http://git.wrl.unsw.edu.au:3000/danh/major-projects-grabber.git
|
|
|
|
|
pip install -e major-projects-grabber
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
## Usage ##
|
|
|
|
|
|
|
|
|
|
### Single modification
|
|
|
|
|
|
|
|
|
|
The URL of each modification page on the DPE Major Projects website ends with a job id, e.g.:
|
|
|
|
|
|
|
|
|
|
majorprojects.planning.nsw.gov.au/index.pl?action=view_job&**job_id=1746**
|
|
|
|
|
|
|
|
|
|
This job id can be used to download all documents associated with a particular modification:
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
|
major_projects_grabber -i 1746
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
### List of search results
|
|
|
|
|
|
|
|
|
|
The DPE portal's search function takes you to a page of search results with a URL like this:
|
|
|
|
|
|
|
|
|
|
http://majorprojects.planning.nsw.gov.au/index.pl?action=search&page_id=&search=&authority_id=425&search_site_type_id=&reference_table=&status_id=&decider=&from_date=&to_date=&x=44&y=10
|
|
|
|
|
|
|
|
|
|
This URL can be used to download all documents for each modifications in the search results page:
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
|
major_projects_grabber -u "http://majorprojects.planning.nsw.gov.au/index.pl?action=search&authority_id=547"
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
### Output directory
|
|
|
|
|
|
|
|
|
|
The output directory to save the downloaded documents can be specified with the `-o` option, e.g.:
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
|
major_projects_grabber -i 1746 -o "DPE documents"
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
If no output directory is specified, the current directory is used.
|
