Simulator GIS – Darb Dabney’s OpenSim

It’s not fully 3D immersive, but hey, 2-1/2D ain’t half bad. The “dsm40cm” model of Marin County has been published as the county’s default terrain on Google Earth. It’s a great pleasure to work with folks who are not troubled by a county representing its surface on a 40cm single-precision float grid that weighs in at 77 GB. In terms of data bulk, that is about the same as the entire 30-meter version of the US National Elevation Dataset.

What one gets when piling that much detail into a single county of around 520 square miles of land area is every building pad, driveway, and crown of road paving that were resolved. The dsm40cm model was developed as an ESRI Terrain Dataset that incorporated our best topographic contours (1:4800 scale 10-foot; 1:2400 scale 2-foot,) photogrammetric break and water lines, FEMA LiDAR and NCALM LiDAR data sets. All told, they currently comprise 40 GB of vector GIS data. Then, breaking the county up into 20 work areas to maintain ArcGIS in a stable and productive state, 40cm posting interval grids were generated that covered the entire county. When finished, these grid tiles were mosaicked with ERDAS Imagine into a single seamless grid. The WGS84 UTM, NAVD88-Geoid 2003 result was provided to the Google Earth team earlier this year.

As with all GIS data sets, it seems, the more detailed it is, the more rapidly it may need updating. In the works for the next year or so are several improvements to the dsm40cm model. First: the photogrammetric break lines will be segregated into steeper sets that tend to run along ridges, and shallower slopes that tend to delineate road cuts and building pads. The ridge set will be used as soft constraints to resolve some artifacts where they rise above some contours.
Second: incorporate new LiDAR data as it becomes available. Some data has already been provided for the lowest part of Lagunitas creek, and it appears that Prof. Ellen Hines of San Francisco State University’s Department of Geography and Human Environmental Studies has been funded by USGS to gather LiDAR county-wide this year.

So there will be revisions, but an exciting aspect is to see data flows being brought into existence that support different levels of mirror world development.
Publishing the dsm40cm model in Google Earth is an important (and beautiful) threshold to cross. Making use of the dsm40cm model in county operations such as creek and watershed delineation will be the practical benefit that drives the work in the first place. And before too many more weeks, there may be entirely new approaches to publishing the data in an immersive environment (neither Second Life nor Opensim) to share.

Kent Woodlands building pad and driveway, in the shadow of Mt. Tam

I’d read about this, but never before experienced the agony first-hand.  Extracting funds from SL, the wait for funds to arrive at PayPal was a bit slow.  In fact, in the time it took funds to go from Linden to PayPal, a bamboo shoot in my back yard could have grown taller than me (that’s my RL not SL height!), and would have been over 2 meters tall.  Anyway, Process Credits are quite lacking in symmetry with how quickly credit charges can flow into the Linden realm.

During this week of waiting my random prims have been cleared out from Amida and nary a trace of Berkurodam BART Station remains besides a video in Gualala.  The video screen was actually entombed by a neighbor, who may not like it but did not send any message.

Anyway–for me this week is all about generating maps and graphics while keeping up with work.  I’ve generated a 50cm terrain grid for parts of my county where perhaps 150,000 people live.  With computational process improvements I should be able to make production stable enough to generate a 25cm grid.  The point is to model terrain slope and aspect within urban parcels.  OpenSim can pack 64 terrain megaprim sculpties over each region to refine terrain more than the built-in 1-meter postings, and display 10cm orthoimagery at full resolution.

Last year, I used first-return LiDAR data of the UC Berkeley campus to generate a 25cm grid for 10cm imagery.  Now, I’m working with bare-earth LiDAR data from FEMA, topographic contours (densified to 1.5m vertex spacing), and most importantly, photogrammetric terrain and water break lines.

Throwing all those data into the mix, the data are built into an ESRI Terrain Dataset, from which I generate TIN and GRID models at various reolution and extent.  The ESRI ArcGIS 3D Analyst Terrain-to-TIN generator breaks down after about 10 mega-faces (so would I…)  And the ArcGIS Terrain-to-GRID generator seems to drift into Windows-unconsciousness after about 1.0 giga-cells.  So for the grid, I break it down and do the pieces, then merge the tiles using ERDAS Imagine, because the ESRI ArcGIS raster mosaic function does not produce output grids much over 10 GB.  As annoying as learning these ArcGIS limits can be, it is very satisfying (and instructive) to see huge swaths of seamless terrain with great detail once it all comes together.  Thanks to the break lines, many driveways and most home building site cuts and fills are resolved.  And it will be a lot of terrain by OpenSim standards–enough to calibrate terrain for over 20,000 contiguous regions–not that I ever expect to build it all at 1:1 scale!

Terrain has been in the mix for me quite a bit these past four weeks.  I’ve worked on pushing ESRI ArcGIS 3D Analyst to its limits of masspoint digestibility, trying hard to bring everything into focus at the same time that everything is sinking down to NAVD88 datum.  An abundant set of waterlines and terrain breaklines have helped to make possible some terrain models that appear to be as good as any one is likely to get from photogrammetric data.  As with LiDAR source, I’m working toward a 30cm gridding interval to sample any reasonable-looking TIN models.

One fascinating aspect of the terrain model is where it ends.  There appears to be a new 1:1200 or 1:4800 shoreline that can be sussed out of some combination of 2.5-foot elevation waterlines, 2-foot elevation contours, and related artifacts.  In fact, it’s a great patchwork of artifacts that must be stringed together.  In the tidal flat areas, there is also plenty of need for validation with multiple photos (hopefully shot at times of lower tides).

Adding to the data bulk there’s a new ortho in town, 30cm natural color flown just about two years ago.  There’s hope of extracting it from the grip of California HARN coordinates after it is all mosaicked.

There’s some progress on a couple of project fronts.  I’ve started assembling some USGS terrain for a 1:10 scale Level 1 build that could involve more OpenSim regions than I’ve ever stood up on one machine before.  Snapshot of progress is here, with a goal of 304 OpenSim regions for the model.  I expect that the OpenSim server will be re-imaged and a new build attempted in the next couple of weeks.

Vicinity of Colorado Springs, CO - with a full mile of terrain elevation subtracted

The site design for the Marin Civic Center build in Second Life Stanford region is also moving along with its target 1/8 region (two SL acres) based on RL terrain and building at 1:1 scale Level 3 build.  Progress sketch below:

Context model data of terrain for Civic Center Administration Building

A bit more can be reviewed by looking at the PDF of the same map here.cc_topo_20090211

Finally, a sky tag has been added to the space above the build. It is visible as a streak to anyone who visits SLURL.com by a mouse roll.  The Stanford region now has a large “MARIN” visible in its upper reach, squarely in the middle of the ancient Second Life Outlands.

Stanford vicinity from SLURL.com on 15 Feb 2009

Today I would like to share the inside production notes (it’s quite low tech for the most part) on making terrain sculpties. I have included a full region’s worth of working raw terrain, and a set of four megaprim sculpties that should help to clarify some of my mutterings in earlier posts. Stuff like precisely which values go into the sculptie gradient maps (shown in a spreadsheet), what it looks like when one takes textures that are 960×960 and add a 64-pixel-extent collar around them, and how to actually configure a region to load and display the 250-meter square at Military Grid Reference System / US National Grid 10SEG_6550_9200, with all the necessary bumpmap and texture files, and instructions for both OpenSim console-side and SL client-side actions.

obdam_40h_php This is how I have created the 40 regions using MGRS / US National Grid naming convention, particularly if the terrain has been scaled to 1.024:1 so that exactly 250×250 meters of RL terrain are loaded into each OpenSim region.

obdf_2_7_f32 This is a single region’s raw float terrain file. The original input digital elevation model had been gridded to have postings every 30cm in X and Y. The nominal scale for OpenSim terrain is 1 meter in X and Y. For large grids of real-world regions, there is reason to scale things up slightly so that there are exactly 16 regions per square kilometer. When one does this, as I have here, the samples are 977 mm and the real-world scale is 1.024:1 or a couple of percent larger than life.

sculpt_gradients_132 is the magic for the sculpties, all one needs to do is take the precise spreadsheet values and create three 8-bit grayscale images from them, using a raster program of your choice, to fit 132×132 size for use as a starting point. Then in the middle 130×130 area of the Z-value image (third or blue channel), insert your 8-bit rescaled values of terrain surface. After that, stack the X, Y, and Z grayscales together as Red, Green, and Blue channels to make a single RGB that will be your UV bumpmap.

Working Example single terrain megaprim bumpmap and texture
This is a single sculptie bumpmap+texture set intended to be placed on a megaprim named ‘nw’ that is sized with ‘edit-scale nw 132 132 164′ on the OpenSim console.

Full 1.024:1-scale region with f32 terrain and four megaprim terrain sculpties This is the real deal, one of the 40 regions in the Open Berkurodam sim and I think it’s an interesting part of its steeper area. The link is to a 10MB zip file that is named for the 250-meter square MGRS/US National Grid region that it represents: 10SEG_6550_9200, the grid point at its southwesterly corner. This archive contains a single-float terrain “obdf_2_7.f32″ raw file ready for loading into an OpenSim region; the file name results from a raster dicing script and this is the second region down and seventh region over from the northwesterly corner of the 40-region sim. The archive also contains four pairs of bumpmap+texture Targa files; their file name results from the same dicing script but the indices are higher because these are quarter-region areas.

10SEG_6550_9200_xml Region configuration file for the following example (I neglected to include it above)

To try out the full region set, take an available OpenSim region, and load in the terrain from the OpenSim console with the following sequence

change-region 10SEG_6550_9200
terrain load obdf_2_7.f32
terrain bake

Next, seed the region with four prims. I tend to fly into the middle of the region or teleport and turn left 90 degrees so I am facing northerly, drop four cubes a few meters apart, then name them by their quadrant: ‘nw’, ‘ne’, ’sw’, and ’se’, ensuring that the Prim’s new names have stuck by checking at least one. Then I fly to the outer edge of the region, or just over into the next southerly one. This is not strictly necessary, but it feels like the right thing to do, sort of like walking a safe distance away after having set four large underground charges. That’s because the next step involves super-sizing. In my lab, the OpenSim server is an Ubuntu Linux box running Mono 1.2.6, but my SL client is on Windows XP, and I switch between machines on a KVM switch going from the OpenSim console to SL client, and it is so much easier to see the megaprims if you aren’t inside them after they have been inflated. On the console:

edit-scale nw 132 132 164
edit-scale ne 132 132 164
edit-scale sw 132 132 164
edit-scale se 132 132 164

The Z value is the one I use in the steepest part of the sim, and this makes some really big cubes. I tend to pull them apart far enough to tell them apart.  Be careful here–I’ve hung a sim that was running fine for a week by planting the seed prims not close to the center and then dragging the centroid of the megaprim over into the next region. Sometimes when handling these megaprims it’s handy to ensure that you have the SL client’s draw distance maxed out to 512 meters, and also to zoom the view out from default one notch with “CTL-8″ to trade of field of view with distance from the prims you are handling. So when you’ve got one of the megaprims selected for editing in the SL client, move them into position by keying locations into the Object tab while being very careful not to touch the values in the Size category (thus saving yourself a visit back on the OpenSim console to reinflate the megaprim) When using but four terrain sculptie megaprims per region, their positions in X and Y are always the same, and the Z position will depend on how you’ve rescaled your floating-point terrain to fit into the 8-bit unsigned approximation. For the example megaprims that I have posted, use these:

Prim ‘nw’ XYZ = 64, 192, 188
Prim ‘ne’ XYZ = 192, 192, 188
Prim ’sw’ XYZ = 64, 64, 189.5 (tweaked for amphitheater vs. region terrain)
Prim ’se’ XYZ = 192, 64, 188

Once I can see that the prims have snapped to fully cover the region and are all nearly the same height, I change their Building Block type to Sculpted and see four really large apples that may be somewhat subterranean. If I hadn’t already done so, I use File > Bulk Upload to get all the bumpmap and texture Targa files into inventory. When you have four megaprim sculpties, you should choose the following for Object/Sculpt Texture and Texture/Texture:

Prim ‘nw’ Sculpt = ‘ob40_03_13_z3.tga’ ; Texture = ‘ob40e_03_13.tga’
Prim ‘ne’ Sculpt = ‘ob40_03_14_z3.tga’ ; Texture = ‘ob40e_03_14.tga’
Prim ’sw’ Sculpt = ‘ob40_04_13_z3.tga’ ; Texture = ‘ob40e_04_13.tga’
Prim ’se’ Sculpt = ‘ob40_04_14_z3.tga’ ; Texture = ‘ob403_04_14.tga’

For the sort of appearance that looks best at first, I have kept the background color in the texture to all 255’s and set the Full Bright to checked. That setting does not do well when it’s night in the sim, but it overcomes some sort of fade that is visible in the texture around the edges of the sculptie when Full Bright is not set. More improvements for the future!

Deep thanks to Adam Zaius for pointing out that I hadn’t really made clear these details in the blog. Enjoy!

The past four days have been a tremendous blur of internalizing NURBS into my mind, at least the SL sculptie variant of them. Now I’ve been aware for several months of how NASA used sculpted prims to represent detailed Mars craters (as published by Ireton), and I’ve certainly followed the beautiful work for David Rumsey done both by Telemorphic in 2003 (3D plots of historic Lake Tahoe area) and more recent historic Yosemite by Nathan Babcock). But there was something confusing and ultimately mysterious about using sculpties for terrain.

Not so much any more. Through several helpful blog and forum posts, and a score of hours spent in experimentation, I feel that I’ve brought the sculptie to heel for my terrain rendering purposes. Mostly, especially for OpenSim, it’s simply to display draped orthoimagery over an already precisely customized region terrain.

What I’ve learned is that for 1:1 mapping, where regional terrain is not available at more detail than the 10-meter postings from seamless.usgs.gov, then one can configure precise sculpted megaprims, only four to a region, and drape imagery quite effectively. The result is real-life imagery draped in the style of Google Earth, but coming out of a free OpenSim server into a free Second Life client, for a dozen or more regions on one server core. When using the technique that I’ve worked out, having only four scuplties to seamlessly cover the region means that the terrain sculpties will rez fully sixteen times (16X) faster than will either the David Rumsey or NASA educational islands.

There’s no special magic here: the region terrain is far superior as a way to represent real life terrain, as it can hold 64K of single-precision floating point values. A sculptie, by comparision, holds a mere 900 usable values that must be compressed to an 8-bit signed integer, for any one of the 900 points’ X, Y, or Z values that are practical to use to guide facets in a terrain “diamond”. This “diamond” is a way of describing what the terrain sculptie looks like after defining the outermost ring of UV values to wrap around to a single point safely below terrain surface, so as not to interfere with the 30×30 values useful to describe terrain in a way that cleanly tiles to cover multiple regions. The vertical scale of the terrain described this way is adjusted with the Z-dimension of the sculptie spheroid, which must be tuned using back-end OpenSim command “edit-scale” if one is manipulating a megaprim.

Anyway, when I get a chance to demo this for some Berkeley terrain, I’ll be sure to post a dramatic screen shot. As it is, the 12-region sim looks so realistic right now that almost any shot would be immediately recognizable by someone familiar with the site, so the good ones will wait for project time. Meanwhile, this one is intended to prove validity of sculpted terrain megaprims for draping orthoimagery.

The tools that I used were: ArcGIS to reproject the seamless terrain and orthoimagery into a rotated local grid variation of WGS84-UTM; ERDAS Imagine to perform mosaicking, dicing, image stretch/rescaling, and layer stacking to build precise UV maps from real life terrain; and OpenOffice.org spreadsheet to calculate precise gradient values for the X and Y components of the UV maps. On the back end was OpenSim 0.5 using region definitions in the 0.4 style, and the terrain build was performed using the standard Second Life 1.19.0.(5) client running on Windows XP with a Radeon X1300 Pro.

A couple of days later, I revisited the sim and made a couple of updates to sculpties with the new 1.19.1.(4) Second Life client, and the orthoimage colors look different depending on sun angle–thanks to Windlight.  It’s not a bad thing, and gives one a reason to look up and appreciate the beautiful sky!   Back on Agni (standard Second Life Grid) by comparison, all the prims seem far more intensely colored and somehow more detailed with the new client.