Simulator GIS – Darb Dabney’s OpenSim

This edition of the county-wide terrain model has been upgraded several ways in terms of input data sets, extent on land, and integration of bathymetric soundings to eliminate coastline clipping.

The LiDAR data set from NCALM / GeoEarthScope has been reprocessed to use inidividual ground-classified returns rather than the 50cm gridded surface; this was made possible by NOAA VDatum, the Java application that has industrial-strength powers to position XYZ files.

Also, NOAA Airborne LiDAR for Assessment of Coastal Erosion (ALACE) data, which are first-return only from 1998–2002, were included where available along the Pacific ocean coastline and edges of some estuaries.

An Airborne 1 LiDAR data set was graciously provided by Marin Municipal Water District where it was available in the lower Lagunitas Creek drainage, and this was used where available and not overlapping with the higher-density NCALM data at Point Reyes Station.

More of the FEMA LiDAR data from Dewberry was included along the eastern, urban floodplain areas. In this terrain model, all FEMA points below 25 meters elevation NAVD88, OR points landing on areas with slope less than or equal to 11 percent, were included. In the 2010.01 terrain, only points below 50 feet NAVD88 were used.

NOAA depth surveys were included with very little filtering, for all nearby soundings since 1931—knowing that tidal bars are dynamic, but including all data as a starting point.

The California Seafloor Mapping Project’s phenomenal multibeam sonar work was incorporated from 2-meter grids offshore to California’s 3-nautical-mile limit, and 1-meter grids within west San Francisco Bay.

The earlier photogrammetric breaklines from VARGIS/Infotech were classified into ridge and road types, and only the road types were retained as hard breakline constraints. Ridge lines and water lines were retained as soft breakline constraints. This has mitigated some of the effects of ridge lines artifacts that derive from inconsistencies between the VARGIS photogrammetry work breaklines and contours, and between VARGIS breaklines or contours and overlapping LiDAR data sets.

Gridding the terrain dataset into a dsm, once again we are flirting with the limits of ArcGIS stability. The processing workstation is imaged with Windows Server 2003 to remove limits on output file size, and to permit reliable killing of wayward processes. (At this time, we’re not certain whether this constitutes a “top kill”, or a “static kill”, but in any case the process ends up terminated ;^)

The 2010.07 edition is a topographic-bathymetric surface model, and its prime use case is the generation of accurate synthetic drainage networks. Together, these features motivate a larger modeled extent than the 2010.01 edition. Along the northern area, the extent was grown to include all watersheds that touch or drain Marin County areas, with some clipping of the easternmost portions of the Petaluma River watershed. Offshore, the legal extent of Marin County guided the clipping area in San Pablo Bay; data extent from CSMP guided eastern limits at Richmond Channel, San Francisco’s North Beach and Presidio shorelines, and westerly from Seal Rock to the 3-nautical-mile limit, then northerly to include all of Bodega Head as imaged by NCALM data.

The extra area has made it infeasible to generate a single 40-cm grid as before, but we have cut the area into quadrants and have generated four tiles as 45-cm gridding. More news soon.

As I learned here, it is even easier than ever for Windows users to browse the Marin dsm40cm terrain.
All those keyboard shortcuts one may have learned to use Google Earth now work in Google Maps with the “Earth” button.
It’s a gas, and all the 40-cm detail is there.

I’ve tested it using Chrome browser on Ubuntu, and the Earth button does not show up.
But using Chrome on Windows XP, it’s there and it works pretty well. No need for composing a web site with the Earth plug-in. It’s just there in the default Google Maps on systems where the plug in would work.

For many of the world’s users of browser-based interactive mapping, integrated 3D viewing has arrived.
Check it out. Link provides an initial point near Civic Center, viewed in Google Earth through Google Maps.

Users who access Google Maps using Windows or Mac should be able to seamlessly launch the Google Earth plugin within Maps.
Just click the Earth button (should be up where the different views are in the upper-right part of the map) if it exists.
If it doesn’t exist, then you are using Linux or a smartphone or are in some other environment that doesn’t support the Earth browser plug-in.

My favorite Google Earth navigation shortcuts:

(Mouse control)
- Drag Mouse regular button to push globe around.
- Release while moving to let globe continue to move with intertia.

Mouse scroll one way or PageUp = pan forward fast
‘+’ = pan forward slowly
Mouse scroll other way or PageDown = pan reverse fast
‘-’ = pan reverse slowly
‘r’ (lowercase R) = return to vertical view, North upward: “get me out of this mess and back to Map view”

(keys used by themselves)
‘w’ or up-arrow-key = spin globe forward
‘a’ or left-arrow-key = spin globe left
‘d’ or right-arrow-key = spin globe right
‘s’ or down-arrow-key = spin globe back

(Ctrl- variants)
(two non-opposing directions at once are allowed)
Ctrl – ‘w’ or up-arrow-key = tilt view upward toward the zenith
Ctrl – ‘a’ or left-arrow-key = pan view left
Ctrl – ‘d’ or right-arrow-key = pan view right
Ctrl – ‘s’ or down-arrow-key = tilt view downward toward nadir

(Shift- variants)
(two non-opposing directions at once are allowed)
Ctrl – ‘w’ or up-arrow-key = orbit point-of-view downward around center view
Ctrl – ‘a’ or left-arrow-key = orbit point-of-view rightward around point in center view
Ctrl – ‘d’ or right-arrow-key = orbit point-of-view leftward around point in center view
Ctrl – ‘s’ or down-arrow-key = orbit point-of-view upward around center view

WILD and FAST
Use Mouse alternate button to free-form navigate
- where you “right” click on the map, a target appears for as long as you keep the alternate button down.
relative to the target:
- drag the cursor up to zoom out
- drag the cursor down to zoom in
- drag the cursor right to rotate view counter-clockwise
- drag the cursor left to rotate view clockwise.

And if everything is just too much, there’s always ‘r’ (lowercase-R) to set everything straight.

Enjoy!

It’s not fully 3D immersive, but hey, 2-1/2D ain’t half bad. The “dsm40cm” model of Marin County has been published as the county’s default terrain on Google Earth. It’s a great pleasure to work with folks who are not troubled by a county representing its surface on a 40cm single-precision float grid that weighs in at 77 GB. In terms of data bulk, that is about the same as the entire 30-meter version of the US National Elevation Dataset.

What one gets when piling that much detail into a single county of around 520 square miles of land area is every building pad, driveway, and crown of road paving that were resolved. The dsm40cm model was derived from an ESRI Terrain Dataset that incorporates our best available topographic contours (1:4800 scale 10-foot; 1:2400 scale 2-foot,) photogrammetric break and water lines, FEMA LiDAR and NCALM (GeoEarthScope) LiDAR data sets. The Terrain Dataset currently comprises 40 GB of vector GIS data.

When the finely detailed surface grids were first developed, we broke the county up into 20 work areas to maintain ArcGIS 9.3.1 in a stable and productive state, and 30cm posting interval grids were generated that covered the entire county–at least during development. When necessary, these grid tiles were mosaicked with ERDAS Imagine into a single seamless grid. The 40cm version was produced directly as a single seamless grid using ArcGIS 9.4 beta 1, on a workstation imaged with Windows Server 2003. The WGS84 UTM, NAVD88-Geoid 2003 result was provided to the Google Earth team earlier this year.

As with all GIS data sets, it seems, the more detailed it is, the more rapidly it may need updating. In the works for the next year or so are several improvements to the dsm40cm model. First: the photogrammetric break lines will be segregated into steeper sets that tend to run along ridges, and shallower slopes that tend to delineate road cuts and building pads. The ridge set will be used as soft constraints to resolve some artifacts where they rise above some contours.
Second: incorporate new LiDAR data as it becomes available. Some data has already been provided for the lowest part of Lagunitas creek, and it appears that Prof. Ellen Hines of San Francisco State University’s Department of Geography and Human Environmental Studies has been funded by USGS to gather LiDAR county-wide this year.

So there will be revisions, but an exciting aspect is to see data flows being brought into existence that support different levels of mirror world development.
Publishing the dsm40cm model in Google Earth is an important (and beautiful) threshold to cross. Making use of the dsm40cm model in county operations such as creek and watershed delineation will be the practical benefit that drives the work in the first place. And before too many more weeks, there may be entirely new approaches to publishing the data in an immersive environment (neither Second Life nor Opensim) to share.

Kent Woodlands building pad and driveway, in the shadow of Mt. Tam

Next week I plan to be on a panel at the Virtual Edge Summit 2010 in Santa Clara on Monday 22 Feb. Inspired by the style of a presentation by State of California GIO Mike Byrne at last month’s BAAMA.org educational session, I will try to prepare an Ignite-inspired talk for Virtual Edge. Since the virtual environment space almost demands it, I will use a film rather than a slide stack to structure my speaking.

The film is mostly cut, and a draft is uploaded in HD – for your perusal. If you want to hear my words try to keep pace with the film, consider attending Virtual Edge Summit, either in person or through one of its virtual channels. Hope to see you there!

If you have the bandwidth, please view the film in HD–I spent most of the past weekend cutting original FRAPS takes at full resolution so that the presentation could stand up to 1080p HD.

Many thanks to Singularity U, director Matt Rutherford, and to Randall Hand who brought it to my attention After chatting at SLCC 2009 this past summer, I appreciate the immediacy of this lecture. OpenSim is discussed around minute 37 (video is available at 720p HD, and is just over 51 minutes long.)
Discussion of augmented reality, and mirror world creation in Second Life and virtual world simulators, just after minute 44.

It’s hard for me to listen to the entire talk just one time and retain the best explanations – but clear and current they are. In a virtual environment, immersed in near-infinite possibilities, Rosedale may no longer be guiding the Second Life ship, but I believe he remains the compass needle

The work on Point Reyes Station has more LiDAR data on tap, but this evening I’m taking a solar sauna in the Kepler planetarium. It’s a captivating display for both young and older eyes.

A large planetarium has our solar system displayed dynamically

Just time to post a couple of shots from Alaska North in ScienceSim.com grid. This build is being created with resources provided by the ScienceSim Land Grant Program, and should remain accessible until June 2010. Downtown area has been refined with 21cm LiDAR grid sculpty prims.
I haven’t figured out how to get the high-res shots with Hippo Viewer, so the 7 Mpel camera in the Linden Lab viewer is still my favorite. Hippo seems to top out at 2 Mpel right now.

Alaska North in Science Sim, featuring 1:4 Point Reyes Station

21cm LiDAR grid (2007) draped with 10cm orthoimagery (2004)

View across Lagunitas Creek toward community of Point Reyes Station

Good stuff has been happening – and GIS data has been finding its way into OpenSim!

Thanks to the persistence of Kim Smith and with help from John Jainschigg, I made my first-ever live audience presentation in the World2Worlds venue on 2009 12 15. Trying to fit everything into a regular work day, I was fortunate to get support to do the presentation from my desk at work rather than from home–and I indulged in a new digital-USB headset for the occasion. Oddly, the headset seemed invisible on the podium ;^)

Speaking at Smarter Technology, 2009 12 15 -- John Jainschigg built a custom podium several steps tall so that people could see my tiny avatar. Photo courtesy of Chimera Cosmos

After nearly a year away from pushing OpenSim (corresponding with my first year on a new RL job), I was a bit nervous about having up-to-date stuff to offer the audience. So instead of a slide stack, I presented the machinima of models that I had brought from OpenSim into Second Life.

Lazy of me, perhaps, but the video did help create a certain party atmosphere in the airy auditorium at W2W. It also helped echo my emphasis on a certain use of OpenSim that I’ve been striving towards since October 2007.
But as so often during RL conferences, the very best part of the presentation experience is the (sometimes serendipitous) human connections that take place around it, and this presentation was no exception to that rule.

One audience member was Richard Hackathorn, who described Second Life in the context of urban planning for listeners to Arina Hadich’s Urban Design Podcast. His enthusiasm for my subject matter at the Smarter Technology presentation has led to an upcoming podcast at UrbanDesignPodcast.com (details to follow). Although I haven’t been a regular listener to podcasts, when the subject of the talks are focused on an area of interest, I am impressed by how much information can be conveyed in the time it takes to listen. The podcast seems to have more detail per minute than I can get reading a browser on my phone, certainly more than I can browse while driving, (and far more detail than I seem to be able to convey in a minute spent writing a blog ;^)

Another audience member was Shenlei Winkler of Fashion Research Institute, who is surely among this world’s most prim-prolific individuals. I’ve been reading about the various Shengri La regions since 2008, created by Shenlei in collaboration with IBM researchers. Ironically, I was collaborating with different IBM researchers about the same time, in 2008 03. These days, Shenlei appears very active with several types of support for the IEEE/ACM-hosted, OpenSim-derived grid known as ScienceSim.com. Shenlei very kindly took time to contact me after the Smarter Technology presentation, introduce me (with voice chat) to ScienceSim and its resident researchers, and encourage me to participate in the Science Sim Land Grant program. More details are described in paragraphs below.

Through ScienceSim, I’ve had the pleasure of interacting with Mic Bowman of Intel Labs, the group managing ScienceSm servers. The way that these folks have configured the servers is glorious. I’ve been a fan of running OpenSim on 64-bit Ubuntu with Mono since 2008 07, and the Intel crew have taken it to such another level that I find it astounding. In ScienceSim, it appears that the ODE physics engine runs for as little CPU cost as that which I’d experienced before with the near-trivial basicphysics. It is a rare treat for me to speak with configurators of OpenSim, much less those who strengthen and extend the simulator code!

Since I was firing up the forges to create some LiDAR sculpties for my ScienceSim project, I decided to warm up the works by creating some carefully-scaled terrain for ScienceSim’s Yellowstone 16-region model. Using public terrain data from the US Geological Survey, I processed terrain down to a model that was 1:83 in the x and y dimensions, and 1:55 in the z dimension (vertical scales are often exaggerated in both hominid avatars and terrain to make them more attractive). As a by-product of the production process, I also saved an intermediate bit of data that I had scaled to 1/10.38 in x and y and 1/6.9 in z. Knowing that it would present a worthy challenge to OpenSim server jocks, (the 1:83 model fit into 16 regions, but the 1:10 model would require exactly 1024 regions) I passed it to Mic along with the 1:83 terrain.

My jaw was somewhat slackened when, less than a week later I heard from Mic that much had been done with the
1:10 terrain. The Intel crew had actually had the tenacity to wait while all 1024 regions were brought up on a single processor; shortly afterward they prudently fit the regions onto a single blade, dual quad-core (Xeons?) system with 11 GB of memory and my favorite X86_64 Ubuntu/Mono environment. As if that wasn’t enough, they did this while configured with ODE physics! I’d say their effort was Olympian, but heck, it actually took them less than six days!

It’s been tremendous fun this month watching OpenSim make bold moves in the direction needed to support the sort of civic paraverse / Immersive Connected Experiences that could back-end many aspects of local agency operations. I’ve been looking for ways to get here since 2006 11, but with Intel’s demonstration of 1K regions on 1 dual-quad-core Xeon blade, I won’t look like such a fool scoping out costs for a county that would require 20K regions to build out at nominal 1:1 scale. (By contrast, in 2007 I had estimated that Linden Lab hosting of 575 regions would cost the City of Berkeley upwards of $60,000 per month!)

The Yellowstone work by Mic has been beautifully documented by Shenlei here, and here and here and here. I’m even using one of Shenlei’s Yellowstone sunset shots as my (Ubuntu) desktop background ;^)

More to the point of my post’s title, I have taken some outstanding publicly available LiDAR data and am in process of crafting a 1/3.9-scale model of Marin County’s own Point Reyes Station, a rural community that is situated so close to the San Andreas Fault that they were imaged in excellent detail as part of a scientific investigation into the geomorphic expression of strike-slip faulting. Together with publicly available high-resolution orthophotography, I’ve been able to refine the technique that was used on the 40-region model of the UC Berkeley campus and develop 21-cm-gridded surface model with 10-cm natural color orthophotography. For a preview of the model at 105-cm surface gridding, view this video. Enjoy, and watch for updates soon.

The Point Reyes Station may be trod upon in ScienceSim’s Alaska North region. Hope you get a chance to visit!

This blog has been shamefully quiet, but Darb’s efforts have been continuing. Just a touch stealthier than before.
On ScienceSim.com, I found a pleasant roost from which to enjoy the solstice sunset.

visiting a rabbit on Shengri La Spirit

Today, the third day of SLCC 2009, Philip Rosedale and Mark Kingdon provided the telephoto and wide-angle views of the road ahead for Second Life.

Union Square as seen from SLCC 2009

Philip summarized his message to a backdrop of an abandoned and decrepit  Detrioit home thus (as best I heard it)
“TRY and recognize that we are at the very beginning, and together, you guys and us, will have to weather a tremendous changing as we move from where we are today
to where this thing is, this kinematic and imaginary global kind of new digital world.”

Mark had a fine intro where he disclosed a moment of discovering his passion for in-world activity while building
“…it was at that point that I got a second island, and the land addiction began.”

Sessions today included morning Art track’s “Plastic Reality”, running over the start of an enterprise session that I really wanted to hear all of: “Enterprise and Virtual Worlds: The Value Proposition” moderated by Dusan Writer.  I stuck around for Schott Homan’s Purdue case study, caught a share of Patio Plasma’s “Building Interactive Science Exhibits, Tools and Techniques”