[1]:
from panel_segmentation.lidar import usgs_lidar_api
import pandas as pd
import os
Jupyter environment detected. Enabling Open3D WebVisualizer.
[Open3D INFO] WebRTC GUI backend enabled.
[Open3D INFO] WebRTCWindowSystem: HTTP handshake server disabled.
USGS LiDAR API#
All USGS LiDAR dataset is pulled via a custom USGS LiDAR API within the panel segmentation package. Using this API, the master_usgs_lidar_dataset_names.parquet and master_usgs_lidar_metadata.parquet are generated and saved in memory within the data folder. This api could also be used to locate and download the USGS LiDAR file that contains the requested latitude-longitude coordinate.
1) Get Master LiDAR Dataset Names#
The master LiDAR Dataset Names has been pulled and saved to master_usgs_lidar_dataset_names.parquet. master_usgs_lidar_dataset_names.parquet contains all the LiDAR dataset project names found within the USGS Projects page on their online database. The parquet file contains columns “dataset_name” and “deprecated_scans”, which flags whether the LiDAR data is an old, deprecated scan.
getAllDataset() function in usgs_lidar_api can be used to repull all LiDAR dataset names in USGS. Please be advised that it may take a couple of hours to pull all the dataset names. Use the saved master_usgs_lidar_dataset_names.parquet instead and read in the parquet file as a dataframe by calling pd.read_parquet().
For this notebook example, one dataset name will be used as an example for pulling the dataset names.
[2]:
# Point to output folder where results are saved
example_output_folder = ("./panel_segmentation/examples/lidar_api_examples")
# Initialize the USGSLidarAPI class
lidar_api = usgs_lidar_api.USGSLidarAPI(example_output_folder)
# Base url of the USGS Projects website
base_dataset_url = "https://rockyweb.usgs.gov/vdelivery/Datasets/Staged/Elevation/LPC/Projects/"
# url path of where the dataset is located
example_current_path = "USGS_LPC_TX_Central_B1_2017_LAS_2019/"
# Get the dataset name for the base url with the example path
dataset_dict, _ = lidar_api.getOneDataset(
current_url = os.path.join(base_dataset_url, example_current_path),
current_path = example_current_path)
dataset_df = pd.DataFrame([dataset_dict])
dataset_df.head(10)
Dataset found: USGS_LPC_TX_Central_B1_2017_LAS_2019/
[2]:
| dataset_name | deprecated_scans | |
|---|---|---|
| 0 | USGS_LPC_TX_Central_B1_2017_LAS_2019/ | False |
2) Get Metadata from Dataset Names#
The master metadata for all LiDAR dataset has been pulled and saved to master_usgs_lidar_metadata.parquet. master_usgs_lidar_metadata.parquet contains the metadata information associated with each .laz file, including its download link and scan boundaries pulled for the dataset names in master_usgs_lidar_dataset_names.parquet. It has columns “dataset_name”, “xml_link”, “laz_link”, “bbox_west”, “bbox_east”, “bbox_north”, “bbox_south”, “start_scan_date”, and “end_scan_date” columns. These metadata information are pulled from an xml file located in the subfolder of the dataset.
Please be advised that it may take a couple of hours to pull the metadata for all the dataset names. Use the saved master master_usgs_lidar_metadata.parquet instead and read in the parquet file as a dataframe by calling pd.read_parquet().
For this notebook example, the example dataset name pulled from the previous cell will be used to show the resultant metadata output.
[3]:
# Get dataset name
dataset_name = dataset_df["dataset_name"].iloc[0]
# Get metadata information from xml files under the dataset name
metadata_df = lidar_api.getAllXmlMetadata(dataset_name,
log_output=False)
metadata_df.head(10)
[3]:
| dataset_name | xml_link | laz_link | bbox_west | bbox_east | bbox_north | bbox_south | scan_start_date | scan_end_date | |
|---|---|---|---|---|---|---|---|---|---|
| 0 | USGS Lidar Point Cloud TX Central B1 2017 stra... | https://rockyweb.usgs.gov/vdelivery/Datasets/S... | https://rockyweb.usgs.gov/vdelivery/Datasets/S... | -97.000314 | -96.984059 | 30.000277 | 29.984160 | 2017/01/28 | 2017/03/22 |
| 1 | USGS Lidar Point Cloud TX Central B1 2017 stra... | https://rockyweb.usgs.gov/vdelivery/Datasets/S... | https://rockyweb.usgs.gov/vdelivery/Datasets/S... | -96.984687 | -96.968430 | 30.000219 | 29.984101 | 2017/01/28 | 2017/03/22 |
| 2 | USGS Lidar Point Cloud TX Central B1 2017 stra... | https://rockyweb.usgs.gov/vdelivery/Datasets/S... | https://rockyweb.usgs.gov/vdelivery/Datasets/S... | -97.000316 | -96.984065 | 29.984579 | 29.968553 | 2017/01/28 | 2017/03/22 |
| 3 | USGS Lidar Point Cloud TX Central B1 2017 stra... | https://rockyweb.usgs.gov/vdelivery/Datasets/S... | https://rockyweb.usgs.gov/vdelivery/Datasets/S... | -96.984691 | -96.968437 | 29.984611 | 29.968493 | 2017/01/28 | 2017/03/22 |
| 4 | USGS Lidar Point Cloud TX Central B1 2017 stra... | https://rockyweb.usgs.gov/vdelivery/Datasets/S... | https://rockyweb.usgs.gov/vdelivery/Datasets/S... | -96.969069 | -96.952809 | 30.000249 | 29.984129 | 2017/01/28 | 2017/03/22 |
| 5 | USGS Lidar Point Cloud TX Central B1 2017 stra... | https://rockyweb.usgs.gov/vdelivery/Datasets/S... | https://rockyweb.usgs.gov/vdelivery/Datasets/S... | -96.953440 | -96.937178 | 30.000278 | 29.984156 | 2017/01/28 | 2017/03/22 |
| 6 | USGS Lidar Point Cloud TX Central B1 2017 stra... | https://rockyweb.usgs.gov/vdelivery/Datasets/S... | https://rockyweb.usgs.gov/vdelivery/Datasets/S... | -96.969065 | -96.952808 | 29.984642 | 29.968522 | 2017/01/28 | 2017/03/22 |
| 7 | USGS Lidar Point Cloud TX Central B1 2017 stra... | https://rockyweb.usgs.gov/vdelivery/Datasets/S... | https://rockyweb.usgs.gov/vdelivery/Datasets/S... | -96.953439 | -96.937181 | 29.984580 | 29.968549 | 2017/01/28 | 2017/03/22 |
| 8 | USGS Lidar Point Cloud TX Central B1 2017 stra... | https://rockyweb.usgs.gov/vdelivery/Datasets/S... | https://rockyweb.usgs.gov/vdelivery/Datasets/S... | -97.000309 | -96.984059 | 29.968971 | 29.952855 | 2017/01/28 | 2017/03/22 |
| 9 | USGS Lidar Point Cloud TX Central B1 2017 stra... | https://rockyweb.usgs.gov/vdelivery/Datasets/S... | https://rockyweb.usgs.gov/vdelivery/Datasets/S... | -96.984685 | -96.968433 | 29.969003 | 29.952885 | 2017/01/28 | 2017/03/22 |
3) Locate the .laz File for a Latitude, Longitude Coordinate#
Using master_usgs_lidar_metadata.parquet, we can get the latest .laz file associated with a latitude, longitude coordinate. This is done by checking if the coordinate is within the scan boundaries of the LiDAR file. If multiple LiDAR files are found, the most recent .laz file is selected. If no LiDAR files are found, then running the LiDAR pipeline to get tilt and azimuth cannot be done. This process is already written in the locateLazFileByLatLon() function in the usgs_lidar_api script. To execute the function, read in the master_usgs_lidar_metadata.parquet file as a Pandas dataframe and provide a latitude, longitude coordinate within the polygon. This function locates the downloadable URL and downloads the .laz file in lidar_laz folder located in the output folder used in the initialization of the class.
For this notebook example, a latitude, longitude coordinate contained within the example dataset will be used. The metadata dataframe is also the Pandas dataframe generated from the example dataset in the previous cell. This metadata dataframe contains the metadata information like the bounding box and a file download link of the scan.
[4]:
# A latitude, longitude coordinate within the dataset
lat, lon = 30.147832489721004, -97.74102015878317
# Locate the .laz file
laz_download_link = lidar_api.locateLazFileByLatLon(metadata_df, lat, lon)
# Download the .laz file
laz_file_path = lidar_api.downloadLazFile(laz_download_link)
print("Download link: ", laz_download_link)
print("Local File Path: ", laz_file_path)
USGS_LPC_TX_Central_B1_2017_stratmap17_50cm_3097513c1_LAS_2019.laz already downloaded.
Download link: https://rockyweb.usgs.gov/vdelivery/Datasets/Staged/Elevation/LPC/Projects/USGS_LPC_TX_Central_B1_2017_LAS_2019/laz/USGS_LPC_TX_Central_B1_2017_stratmap17_50cm_3097513c1_LAS_2019.laz
Local File Path: ./panel_segmentation/examples/lidar_api_examples\lidar_laz\USGS_LPC_TX_Central_B1_2017_stratmap17_50cm_3097513c1_LAS_2019.laz