Darb Dabney's 3d GIS – OpenSim

The manual process I’ve been using to place the terrain megaprims is crying out for automation, but for the moment, I’ve added it only incrementally. I have three OpenSim command scripts now for the three scales of terrain megaprims, and these use edit-scale to supersize the seed prims that I am still putting, four to a reigon, in starter positions by hand.

I’ve had some trouble with my SL client not always showing me all items in inventory. Also, I fixed the one problem terrain megaprim. Apparently it was a naming convention error from when I created the three sets of different-scale terrains as 8-bit, and the problem prim was where I first realized that ERDAS Imagine dicing was starting the tile names anew for each of the rescales, and this was confusing as all three had different upper-left origins.

Even so, I’m only up to having 19 reigons populated with surface terrain sculpties. It is rather satisfying when the megaprims get sized as cubes, positioned appropriately in 3-space, then converted to giant generic sculpties—and when the proper terrain UV is applied, they just snap to the terrain like a fairly decent coat of paint the instant that I select the next prim.

The imagery is tiling very well and that is also satisfying, when I turn on Full Bright and see good detail. It seems that the key to perfect tiling is to adjust the leftmost texture tiles to have a -0.064 horzontal shift to their 1.000 scale texture, and the topmost texture tiles have an 0.064 vertical shift. The upper-leftmost tile has both. I even found a farm in Berkeley–go figure. Looks like they were growing strawberries near Oxford and Hearst on 1 July 2006.

Not to be deterred by having everything going well, however slowly, I opted to test the new distribution of Ubuntu tonight. My “save-xml” scripts ran without warning, but following the OS upgrade, and verification of graphics and browser, my load-xml script only place about one good megaprrim per region. I’ll dup them and get back on track soon. The browser (Swiftweasel) had nary a hiccup, and still runs Flash just fine. My Nvidia display driver kept on working, and I did not need to re-install it as I had been warned when first installing the driver by Envy. The default desktop has a cool Heron on it, the System Monitor has transcended its previous Windows-like aesthetic and improved the dashboard. MySQL seems to be unaffected by the upgrade. Eclipse Europa launches in about 15 seconds. Things feel just a bit faster in many departments, including the rate at which OpenSim draws land.

What’s next:
1) I’d like to finish my way through building the 40-region model on the local OpenSim server here in the lab
2) I’d like to craft a 1:25 scale model of the entire 160-sculptie terrain on Agni in a parcel on Amida not far from to the Berkurodam BART Station 1:3 model.
3) if Hardy Heron holds together for the next few days of testing, I’d like to build a public-facing version of the OpenBerkurodam model, at least as a standalone sim

.

There has been a bit of head scratching as other distractions apparently clouded an obvious scale issue. The first terrain megaprim sculptie project done last month, had available imagery at 30cm.  For that, it only made sense to oversample to 25cm to make 512×512 textures.  That decision led to my adding a collar around the original 512′s until they clicked into the proper size without rescaling on a quarter-region megaprim. With Berkeley, the source imagery is almost 10cm (103mm pixels) and the challenge has been to size the resample so as to make best use of the 1024×1024 texture size limit per prim.

Where I took a wrong turn was trying to proportion the collar that was added to the 512′s, rather than going back to basic principles with sculpties. Bottom line: my efforts of the past week went astray as I allowed confusion to set in, casting about for the proper maximum texture dimensions working down from 1024. (and I’ve got the awkward attempts at 1008, 994, and 978 pixels to prove it).

In fact, the answer is very simple in reference to basic sculptie principles, as the maximum dimensions of the sculptie bumpmap are 32×32, and due to the need to wrap it around to an apex underneath, this can only represent a 30×30 terrain patch. Thus, the maximum imageable area is simply (30/32)*1024, or 960 pixels square, collared out to 1024 square to make each orthophoto tile. This means that an OpenSim 1.024:1 model can accomodate 130mm orthophoto imagery, and I now have 160 tiles ready to go with the bumpmaps.

So far I’ve configured twelve regions with their megaprims, and only one seems to have issues with the height of the sculptie to stay 30cm afloat the terrain surface. Nine of these reigons use the flattest setting, one uses the intermediate, and two use the steepest. Here’s some shots for update’s sake. The full set of orthophoto textures have been uploaded (450 MB of Targa files) and seem to show up reasonably well in inventory. I am using a local MySQL instance on the OpenSim machine for prim storage.

After spending plenty of time getting all the terrain megaprims stamped out, and starting some refinements of how to squeeze the imageable part of the ortho into a portion of the 1024 square texture, I found myself rather unhappy with how lumpy the megaprims were in the flat part of the model. Berkeley has this sort of dual terrain personality (no comment on the residents) that has certain types of details in the distal fan and floodplain parts of town to the westerly, and very different types of details in the hilly and steep areas.

When one loads the natural terrain into OpenSim, the values are 64K of single-precision floating points per region. When loading terrain into a megaprim, there are a mere 900 usable values that must be mashed into an 8-bit integer of Z values. So when the whole sim ranges from 45–267 meters (and it could go up to 581 meters if a sim ran all the way up to Vollmer Peak) , one gets dynamic range issues if all the terrain megaprims are scaled to fit over the entire sim. So to mitigate this, I divided the sim into distal (the “flats”), proximal (the “foots”) , and hills proper (easterly of the Hayward Fault).

The upside of this extra work is better fidelity in the different parts of the model when the floating point values are approximated by 8-bit integers. When I used the entire elevation range for the whole UC Berkeley sim, each integer Z value was 87 cm, or close to three feet. With the sim broken into three regions of megaprim Z-scaling, I have each integer worth 11 cm in the distal, 31 cm in the proximal, and 63 cm in the hills, so everything is a little better everywhere. I’ll fess up to not having the prim placement fully automated, otherwise it would be practical, and perhaps desirable, to use the full dynamic range over just those elevation values in the region (or even in the quadrant of the region that the terrain megaprim covers). But by that point, I’d consider even denser grids of terrain megaprims, and it would then be a different process for representing the terrain.

After all, the whole point of this exercise is to devise a work-around for not being able to load the ortho imagery directly into the region as a draped terrain texture!

Enough blabbing – please enjoy the graphics.

Here is how the terrain maps out versus the buildings

Not intended as digital Cubism, this odd-looking approach makes better terrain megaprims. Really, it does!

In case one knows particular buildings on the UC Berkeley campus or environs, this is how the zoning worked out. There is a certain logic to it, geomorphically.

I’ve been in a bit of a rut the past couple of days, feeling doubt about which way to proceed with configuring the OpenSim side of the UC Berkeley campus 1.024:1 sim. For the first time since I started setting up OpenSim test servers back in October 2007, I was uncertain of my ability to make it work with this project. I rolled back to 0.4, 0.5.0, 0.5.1, and the trunk that worked a couple of days ago would run only 32 regions well, and even at that would stop working, without any use, by morning. All my effort was going into testing out various ways of retreating from the leading edge. In an activity like OpenSim, that’s not a fun place to toil!  Now, after the sim sits quietly through the night, I can teleport from my landing zone in the far SWly region to the far NEly region, and get there pronto.  Plus the 40 Regions are barely consuming 1% of the CPUs.

Realizing that a good 48 hours had passed, one of the things I tried tonight was a fresh grab of the trunk, and that really turned things around for me. With OpenSim 0.5.4_4272 I have the same rocket-fast launch, zippy association of terrain with regions, and I can actually teleport into regions that haven’t resolved their terrain without finding my av hung up. That was all good. Then, I started moving around the ERDAS Imagine data that will be stamped into terrain megaprims, and I was reminded that I’d gone to all the trouble of resampling both terrain and orthoimage for 40 regions, and my diced file naming conventions were already dependent on that entire set of 5 x 8 regions. So rather than fire up the process for making sculptie bump-maps, I went back to the 0.5.4_4272 build, shut it down and went after my region configuration–willing to give it another try at 40 regions. While I was at it I generalized my PHP region XML configurator.

Just for the sake of enjoyment, here are a few views, including nice Windlight night shots. Compared to two days ago, one can notice more of the hill at LBNL, with the fairly intricate grading for service roads and laboratory buildings quite evident in the scene.

Enjoy!

Somehow, things are harder at full scale than they were just a few months ago doing 1:4 work in OpenSim 0.4 for all of Berkeley. The new OpenSim 0.5.4 has some aspects, like initial launch, that are blazingly fast. Still, I struggled with what surely acts like lazy instantiation in the new code. It launches like a rocket, but when I touch it with a client, lots of the same old slogging starts to happen. Very much most significantly for my extensive and largely static work, the current 0.5.4 will not configure stably for more than 35 regions on one (dual-core) server, even with BasicPhysics.

So, after fighting with it for three days, rolling back toward 0.5.0, and finally flopping down to tonight’s SVN trunk, I have reached a spatial compromise that can be shared. I really wrung my heart out trying to pare down the 40-region design posted last week into an “essential” 35 or fewer regions, balancing the virtues of demonstrating 1:1 scale paraverse work in a relevant way to students, Cal faculty, and City government interests. After hours of exploring options, I threw out the northernmost eight regions, to leave a 4×8 array of 32 regions that retain (somewhat selfishly) the Martin Luther King Jr. Civic Center Building, as well as the California Memorial Stadium (for Hayward Fault interests), the Greek Theater, and the Berkeley Community Theater (for musical memories’ sake).

The terrain is of the finest quality for use in OpenSim. I took Alameda County LiDAR-based bare-earth terrain in triangulated irregular network (TIN) form, and using the ArcGIS 9.2 3D Analyst extension, ground out a 30-cm posting interval grid version, in WGS84 UTM 10 north meters, NAVD88 Geoid2003 meters, for processing in Leica Geosystems ERDAS Imagine 9.1 to produce diced tiles that were flipped and exported to raw single-precision floating point, byte-swapped (Motorola style) raw binary terrain tiles that OpenSim so readily digests. To implement my chosen 1.024:1 well-tempered scale, each “1-meter” terrain sample in the sim was actually 976.5 mm in sample interval, simply taking 256 samples over 250 meters of real-world terrain for each sim X- and Y- axis. Thus far, I have neglected to scale the Z in the same way, but when I get to stamping out the terrain megaprims, and need to rescale the terrain for best fit, I’ll rescale.

Here’s two views of the sim running on a dual-core 3.4 GHz server (Ubuntu 7.10 / Mono). One view from the Synchotron building area of Lawrence Berkeley National Lab, at the northeasterly corner of the sim, looking southwesterly.

The other view shows how incredibly detailed and appropriate the LiDAR terrain is when used in 1:1 scale OpenSim application. The view is from near the SEly corner of the sim, and shows the bare-earth expression of the California Memorial stadium, the Greek Theater, and Bancroft Ave running westerly, as visible by its smooth, crowned road grade and gutters, all plainly visible in the terrain.

Enough carefree hours in the main SL Agni grid, already! Back to matters of creation.

Next big thing should be a terrain prototype for civic application. No special business process in mind here, just a demo of the draped imagery on real-life terrain in a way that could scale up city-wide. For starters, there must be a better correspondence between the US National Grid and the dimensions of the simulated region. Sure Neal Stephenson may have suggested binary 2^n dimensionality, and there may be plenty of reasons in the simulator code to make use of the full range of 256 meters. But after more than a handful of regions, the starting corners get downright ugly.

So I won’t do it that way. By scaling up, larger even than real-life, the regions can be built sixteen-to-a square kilometer. In a worldwide sense, except for the matter of 62 or 64 matchlines, the US National Grid (a.k.a. Military Grid Reference System or MGRS) has the whole world in its hands, so harmonizing region design with that grid plan covers a whole lot of ground. To minimize my effort at constructing regions, while planning for worldwide sim grid extensibility, I have chosen to configure the overall sim to represent 250-meter square patches of real earth using each of its 256-meter square regions.

This scales the real-world up a shade in the sim, to (1.024 : 1) but allows every fourth region in X and in Y to start on an exact grid kilometer. That scale produces 16.0000 regions per square kilometer, rather than 15.2588 regions/square km. From the geography side of things, this harmony is attractive since every fourth region will snap to a grid kilometer instead of every 1000th region. Even at that, the grid kilometer that 1000 of those 256 meter regions snap to is 256 kilometers, which is much clumsier to locate by name.

Thus the “well-tempered” moniker for this scale is well deserved, as any real-world USNG/MGRS grid coordinate could then be used to search for the relevant simulator region from a moderately simple bit of string manipulation. For Berkeley, and the western part of California, the zone is “10S” and the 100-kilometer grid within that is “EG” for San Francisco and Berkeley area. Put together, the US National Grid designator for the 100-km square is sometimes called “10SEG”, depending on where folks do or don’t put spaces.

If we always have exactly sixteen (16) regions per square kilometer, then we can use the shorthand version of the USNG grid names that only detail down to 10-meter increments. In this way, a region with its southwesterly corner at WGS84 UTM zone 10 north, 564000 meters Easting, 4191250 meters Northing, can have the US National Grid 10-meter designation of “10SEG 6400 9125″, which could be mashed together without spaces, or used to name a simulator region such as “10SEG_6400_9125″ in a slightly more readable form. For those of us who no longer have youthful eyes, the tiny little display on the Second Life client for the region name motivates the use of spaces.

So here’s a graphic of the plan: a 40-region prototype (5 x 8 regions), which will be configured with only Basic Physics, but real-life LiDAR-based terrain, and four megaprim sculpties per region to drape imagery (10 x 16 terrain sculpties) such as 10cm natural color orthophotography. Here’s where I hope to take this:

Just a note to describe more fun sailing with the new Windlight viewer, and the Wee Tiny Tako 3.2 (that’s a version, not its length. The actual craft is sporting a 2.25-meter waterline!

For my Dad’s birthday this week, I took a sail to the Northern Continent.

Along the way I sailed through ANWR
Finally, I met some unpleasantly private waters that were poorly marked, and my craft reverted to my Inventory, leaving me sitting on a coral reef. Just before this mishap, I enjoyed some fresh winds up to 12 knots northerly and boat speeds up to 7 knots.

Plenty of fun!

This post has little to do with OpenSim, except to share the enjoyment of the new viewer that works well enough with sculpted terrain megaprims in OpenSim. This evening was simple diversion, a novel experience of sailing in a Wee Tiny Flying Taco, learning how to sail it better, and navigating the ancient inland seas from Gualala through Rosedale and Kapor to the timeless shores of DaBoom. Virtual weather was very pleasant with a westerly 7 knot breeze. I moored off Stanford’s southern shore and strolled a bit, and got to see some square-rigged pirate ships up close. The tiny Taco fits well under bridges and as always, Second Life from tiny eyes seems bigger and perhaps more wondrous.

One annoyance, for me, was that the new viewer seems to have altered the rules for focus-and spin when mousing. While learning how to build detailed interiors of structures, I really got used to the click-to-focus, followed by an Alt-click to rotate point-of-view around that clicked focus point. Now that seems lost and I miss it very much. It matters not too much with sailing, but the way it works now would seem crippling to me when it came time to build fancy structures. So for that, maybe OpenSim folks will want to keep the 1.19.0.5 installer fairly close at hand!

The past four days have been a tremendous blur of internalizing NURBS into my mind, at least the SL sculptie variant of them. Now I’ve been aware for several months of how NASA used sculpted prims to represent detailed Mars craters (as published by Ireton), and I’ve certainly followed the beautiful work for David Rumsey done both by Telemorphic in 2003 (3D plots of historic Lake Tahoe area) and more recent historic Yosemite by Nathan Babcock). But there was something confusing and ultimately mysterious about using sculpties for terrain.

Not so much any more. Through several helpful blog and forum posts, and a score of hours spent in experimentation, I feel that I’ve brought the sculptie to heel for my terrain rendering purposes. Mostly, especially for OpenSim, it’s simply to display draped orthoimagery over an already precisely customized region terrain.

What I’ve learned is that for 1:1 mapping, where regional terrain is not available at more detail than the 10-meter postings from seamless.usgs.gov, then one can configure precise sculpted megaprims, only four to a region, and drape imagery quite effectively. The result is real-life imagery draped in the style of Google Earth, but coming out of a free OpenSim server into a free Second Life client, for a dozen or more regions on one server core. When using the technique that I’ve worked out, having only four scuplties to seamlessly cover the region means that the terrain sculpties will rez fully sixteen times (16X) faster than will either the David Rumsey or NASA educational islands.

There’s no special magic here: the region terrain is far superior as a way to represent real life terrain, as it can hold 64K of single-precision floating point values. A sculptie, by comparision, holds a mere 900 usable values that must be compressed to an 8-bit signed integer, for any one of the 900 points’ X, Y, or Z values that are practical to use to guide facets in a terrain “diamond”. This “diamond” is a way of describing what the terrain sculptie looks like after defining the outermost ring of UV values to wrap around to a single point safely below terrain surface, so as not to interfere with the 30×30 values useful to describe terrain in a way that cleanly tiles to cover multiple regions. The vertical scale of the terrain described this way is adjusted with the Z-dimension of the sculptie spheroid, which must be tuned using back-end OpenSim command “edit-scale” if one is manipulating a megaprim.

Anyway, when I get a chance to demo this for some Berkeley terrain, I’ll be sure to post a dramatic screen shot. As it is, the 12-region sim looks so realistic right now that almost any shot would be immediately recognizable by someone familiar with the site, so the good ones will wait for project time. Meanwhile, this one is intended to prove validity of sculpted terrain megaprims for draping orthoimagery.

The tools that I used were: ArcGIS to reproject the seamless terrain and orthoimagery into a rotated local grid variation of WGS84-UTM; ERDAS Imagine to perform mosaicking, dicing, image stretch/rescaling, and layer stacking to build precise UV maps from real life terrain; and OpenOffice.org spreadsheet to calculate precise gradient values for the X and Y components of the UV maps. On the back end was OpenSim 0.5 using region definitions in the 0.4 style, and the terrain build was performed using the standard Second Life 1.19.0.(5) client running on Windows XP with a Radeon X1300 Pro.

A couple of days later, I revisited the sim and made a couple of updates to sculpties with the new 1.19.1.(4) Second Life client, and the orthoimage colors look different depending on sun angle–thanks to Windlight.  It’s not a bad thing, and gives one a reason to look up and appreciate the beautiful sky!   Back on Agni (standard Second Life Grid) by comparison, all the prims seem far more intensely colored and somehow more detailed with the new client.