Darb Dabney’s OpenSim Blog

I’ve been in a bit of a rut the past couple of days, feeling doubt about which way to proceed with configuring the OpenSim side of the UC Berkeley campus 1.024:1 sim. For the first time since I started setting up OpenSim test servers back in October 2007, I was uncertain of my ability to make it work with this project. I rolled back to 0.4, 0.5.0, 0.5.1, and the trunk that worked a couple of days ago would run only 32 regions well, and even at that would stop working, without any use, by morning. All my effort was going into testing out various ways of retreating from the leading edge. In an activity like OpenSim, that’s not a fun place to toil!  Now, after the sim sits quietly through the night, I can teleport from my landing zone in the far SWly region to the far NEly region, and get there pronto.  Plus the 40 Regions are barely consuming 1% of the CPUs.

Realizing that a good 48 hours had passed, one of the things I tried tonight was a fresh grab of the trunk, and that really turned things around for me. With OpenSim 0.5.4_4272 I have the same rocket-fast launch, zippy association of terrain with regions, and I can actually teleport into regions that haven’t resolved their terrain without finding my av hung up. That was all good. Then, I started moving around the ERDAS Imagine data that will be stamped into terrain megaprims, and I was reminded that I’d gone to all the trouble of resampling both terrain and orthoimage for 40 regions, and my diced file naming conventions were already dependent on that entire set of 5 x 8 regions. So rather than fire up the process for making sculptie bump-maps, I went back to the 0.5.4_4272 build, shut it down and went after my region configuration–willing to give it another try at 40 regions. While I was at it I generalized my PHP region XML configurator.

Just for the sake of enjoyment, here are a few views, including nice Windlight night shots. Compared to two days ago, one can notice more of the hill at LBNL, with the fairly intricate grading for service roads and laboratory buildings quite evident in the scene.

Enjoy!

Enough carefree hours in the main SL Agni grid, already! Back to matters of creation.

Next big thing should be a terrain prototype for civic application. No special business process in mind here, just a demo of the draped imagery on real-life terrain in a way that could scale up city-wide. For starters, there must be a better correspondence between the US National Grid and the dimensions of the simulated region. Sure Neal Stephenson may have suggested binary 2^n dimensionality, and there may be plenty of reasons in the simulator code to make use of the full range of 256 meters. But after more than a handful of regions, the starting corners get downright ugly.

So I won’t do it that way. By scaling up, larger even than real-life, the regions can be built sixteen-to-a square kilometer. In a worldwide sense, except for the matter of 62 or 64 matchlines, the US National Grid (a.k.a. Military Grid Reference System or MGRS) has the whole world in its hands, so harmonizing region design with that grid plan covers a whole lot of ground. To minimize my effort at constructing regions, while planning for worldwide sim grid extensibility, I have chosen to configure the overall sim to represent 250-meter square patches of real earth using each of its 256-meter square regions.

This scales the real-world up a shade in the sim, to (1.024 : 1) but allows every fourth region in X and in Y to start on an exact grid kilometer. That scale produces 16.0000 regions per square kilometer, rather than 15.2588 regions/square km. From the geography side of things, this harmony is attractive since every fourth region will snap to a grid kilometer instead of every 1000th region. Even at that, the grid kilometer that 1000 of those 256 meter regions snap to is 256 kilometers, which is much clumsier to locate by name.

Thus the “well-tempered” moniker for this scale is well deserved, as any real-world USNG/MGRS grid coordinate could then be used to search for the relevant simulator region from a moderately simple bit of string manipulation. For Berkeley, and the western part of California, the zone is “10S” and the 100-kilometer grid within that is “EG” for San Francisco and Berkeley area. Put together, the US National Grid designator for the 100-km square is sometimes called “10SEG”, depending on where folks do or don’t put spaces.

If we always have exactly sixteen (16) regions per square kilometer, then we can use the shorthand version of the USNG grid names that only detail down to 10-meter increments. In this way, a region with its southwesterly corner at WGS84 UTM zone 10 north, 564000 meters Easting, 4191250 meters Northing, can have the US National Grid 10-meter designation of “10SEG 6400 9125″, which could be mashed together without spaces, or used to name a simulator region such as “10SEG_6400_9125″ in a slightly more readable form. For those of us who no longer have youthful eyes, the tiny little display on the Second Life client for the region name motivates the use of spaces.

So here’s a graphic of the plan: a 40-region prototype (5 x 8 regions), which will be configured with only Basic Physics, but real-life LiDAR-based terrain, and four megaprim sculpties per region to drape imagery (10 x 16 terrain sculpties) such as 10cm natural color orthophotography. Here’s where I hope to take this:

The past four days have been a tremendous blur of internalizing NURBS into my mind, at least the SL sculptie variant of them. Now I’ve been aware for several months of how NASA used sculpted prims to represent detailed Mars craters (as published by Ireton), and I’ve certainly followed the beautiful work for David Rumsey done both by Telemorphic in 2003 (3D plots of historic Lake Tahoe area) and more recent historic Yosemite by Nathan Babcock). But there was something confusing and ultimately mysterious about using sculpties for terrain.

Not so much any more. Through several helpful blog and forum posts, and a score of hours spent in experimentation, I feel that I’ve brought the sculptie to heel for my terrain rendering purposes. Mostly, especially for OpenSim, it’s simply to display draped orthoimagery over an already precisely customized region terrain.

What I’ve learned is that for 1:1 mapping, where regional terrain is not available at more detail than the 10-meter postings from seamless.usgs.gov, then one can configure precise sculpted megaprims, only four to a region, and drape imagery quite effectively. The result is real-life imagery draped in the style of Google Earth, but coming out of a free OpenSim server into a free Second Life client, for a dozen or more regions on one server core. When using the technique that I’ve worked out, having only four scuplties to seamlessly cover the region means that the terrain sculpties will rez fully sixteen times (16X) faster than will either the David Rumsey or NASA educational islands.

There’s no special magic here: the region terrain is far superior as a way to represent real life terrain, as it can hold 64K of single-precision floating point values. A sculptie, by comparision, holds a mere 900 usable values that must be compressed to an 8-bit signed integer, for any one of the 900 points’ X, Y, or Z values that are practical to use to guide facets in a terrain “diamond”. This “diamond” is a way of describing what the terrain sculptie looks like after defining the outermost ring of UV values to wrap around to a single point safely below terrain surface, so as not to interfere with the 30×30 values useful to describe terrain in a way that cleanly tiles to cover multiple regions. The vertical scale of the terrain described this way is adjusted with the Z-dimension of the sculptie spheroid, which must be tuned using back-end OpenSim command “edit-scale” if one is manipulating a megaprim.

Anyway, when I get a chance to demo this for some Berkeley terrain, I’ll be sure to post a dramatic screen shot. As it is, the 12-region sim looks so realistic right now that almost any shot would be immediately recognizable by someone familiar with the site, so the good ones will wait for project time. Meanwhile, this one is intended to prove validity of sculpted terrain megaprims for draping orthoimagery.

The tools that I used were: ArcGIS to reproject the seamless terrain and orthoimagery into a rotated local grid variation of WGS84-UTM; ERDAS Imagine to perform mosaicking, dicing, image stretch/rescaling, and layer stacking to build precise UV maps from real life terrain; and OpenOffice.org spreadsheet to calculate precise gradient values for the X and Y components of the UV maps. On the back end was OpenSim 0.5 using region definitions in the 0.4 style, and the terrain build was performed using the standard Second Life 1.19.0.(5) client running on Windows XP with a Radeon X1300 Pro.

A couple of days later, I revisited the sim and made a couple of updates to sculpties with the new 1.19.1.(4) Second Life client, and the orthoimage colors look different depending on sun angle–thanks to Windlight.  It’s not a bad thing, and gives one a reason to look up and appreciate the beautiful sky!   Back on Agni (standard Second Life Grid) by comparison, all the prims seem far more intensely colored and somehow more detailed with the new client.

Over the past four weeks I’ve participated in a burst of effort that has involved multiple resolutions of GIS data to describe a campus terrain in an eastern US state. It’s been over twelve years since I’d worked with a 3D CAD design for a slope design, and this one brought back memories and put detail into a much finer scale, working with 30cm contours around one new building, where my earlier efforts were more like 2m contours over a large industrial site. All the same, having once “written” a slope design, I found it possible to “read” the design work that existed in various layers, some 3D and some 2D, in a new site design.

One of the beautiful and satisfying aspects of GIS work can be the integration of disparate data sets into a spatially coherent whole. In this case, I sought to make an OpenSim terrain that would be home to a very complicated build where two CAD drawings held structure details, a third held the site and design terrain, and as is often the case, a merge with surrounding seamless USGS DEMs was desired.

The first step was to ensure that the site base map and structure plans matched scale and could be lined up. A combination of grounding mat and retaining walls in the site plan made that possible, and in ArcGIS, the AutoCAD DWGs for the structure were translated onto the site plan drawing. That was a lot of work, with over 2Meg of polylines in the structure, and I knew from my Berkurodam build that keeping a complex structure square with the region grid would be frequently appreciated by those working this build.

To earn the appreciation of the build workers while merging the region with the surrounding world, it was necessary to define a projection-referenced local coordinate system for the campus. Mercifully, the site plan contained many building footprints across the street from the construction site, and these were easily identified in publicly available regional 30cm orthoimagery. I chose a well-defined corner of one building as my pivot point, calculated both WGS84 UTM meters and local campus grid inches (converted to meters) for that point, and defined grid north as 014 degrees azimuth on the UTM grid.

The facility for defining a local grid this way is easy in ESRI ArcGIS, and with it I could create a Feature Dataset dimensioned in meters for the campus grid. I was able to export the CAD drawing polylines into an inches-denominated version of the campus grid, then select a fistful of layers and export those into the meters-denominated campus grid for editing and 3D terrain work. It was especially helpful to have ArcGIS’s ability to project the UTM-based orthos into the local grid, as I was not able to figure out how to include an arbitrary grid rotation in an ERDAS custom projection. Once I knew that I had a projection-based local grid definition and could move GIS features and imagery back and forth from UTM, I imported two 1:24,000 sheets’ worth of 10-meter DEM data, about 6km by 7km of 30cm natural color imagery, and I was comfortable diving into the grading plan model knowing that regional context would be available.

The greatest labor involved reading the site plan drawing in detail, clipping out regraded areas from the existing terrain which was a 3D drawing, then editing each segment of each contour in the grading plan (sadly a 2D drawing) to have the proper elevation attribute so that it could flow into the existing terrain contour. A lot of this took careful contour reading and switching views between the multilayer DWG view in one ArcMap session (where I could read all the annotation like contour labels), and the imported polyline contours being edited sans annotation in another ArcMap session (where I had not had success importing the CAD annotation). One of my first realizations when I started keeping two ArcMap sessions running (to keep an eye on the CAD annotation) was that I had misread the drainage swales and had grown them out of the terrain, placing drainage lines farther upslope that was actually in the design.

After a couple of weeks calendar time, I had completed what might be the first 3D CAD model of the site as it was designed. At least, if a 3D model already existed, I did not have access to it. When every segment of every contour had the proper elevation attribute, I regenerated the 3D polylines with Z values only from the attributes. Then using typical techniques, I created a TIN with hard breaks on the contours, and ground out both 1-meter and 0.5-meter postings of gridded terrain from the TIN. This provided highly detailed terrain for at least one OpenSim region.

Merging this 1-meter posting terrain with 10-meter regional data that had been rotated 14 degrees was the next challenge. First, I made an effort to harmonize the vertical datums. The regional DEMs had typically excellent metadata and with very little interpretation, made it possible to define them as WGS84 NGVD29 meters, which I recalculated to WGS84-Geoid2003 NAVD88 meters. The CAD site plan fit most closely with the resulting grid when I just defined it as WGS84-Geoid2003 NAVD88 meters (from design Z-inches), although it remains about 1.5 meters lower at its edges than regional DEM, when overlaid on the regional data.

The typical challenges of oversampling coarse grids were quite graphic in the regional data. After using ArcGIS to project a WGS84 UTM copy of the regional DEM into the campus grid, I used ERDAS Imagine to resample that 10-meter source into a 1-meter working model. Having 100 grid cells with identical elevation looked unnatural at best, so I used an ERDAS focal analysis with a 7×7 kernel to estimate a mean value for every cell in the 1-meter resampled version of the local-grid-projected version of the regional DEM. While normally this would be an extreme focal analysis, it seemed to produce a reasonable result for the 10×10 oversampled regional DEM.

When the regional grid was prepared, I used ERDAS mosaicking to overlay the detailed site plan, and then found a desirable origin point and clipped a subset 768m by 1024m area that, based on the co-registered 30cm ortho imagery, just covered the campus area with 1-meter posting terrain and could be stood up as 12 OpenSim regions at 1:1 scale.

For texturing purposes, I used ERDAS to resample the campus grid version of the orthos that had been reprojected by ArcGIS, from the original 30cm to 25cm pixels. This allowed clipping of a subset of the model area as a 3072 x 4096 image, providing a 1024×1024 texture for each OpenSim region after dicing. For demonstration purposes, and as a guide to build-out of less detailed structures around the campus, these ortho tiles are used to texture on a 256m by 256m by 0.4m horizontal megaprim plate centered on each region. These textured megaprim plates make it easy to calculate sim region coordinates for buildings and objects visible in the ortho imagery.  These calculations should be useful when synchronizing real-world activity with its simulator facsimile.

Don’t fret about the silence here of the past two months–activity in the lab has been greater than ever before!
The 1 GHz Coppermine PIII / 1.5 GB memory has had 81 sims sqeezed onto it (with mere Basic Physics), and has been tested with three users, loaded with real-life terrain, and offshore areas filled with orthoimage-decked megaprims.

Really - please check out the new screenshots posted on OpenSimulator.org

More, there’s a new system on shakedown. It’s an ASUS P5KC, with Core2 Duo E6550 overclocked to 3.4 GHz with 4 GB DDR2800 overclocked to 485 / 970 MHz. Ubuntu 7.10 Gutsy x86_64 is getting decked out with x64 VMware server, Samba 4 / AD Domain Controller, and soon will check out how far OpenSim can get scaled up from 1:4 closer to 1:1 with these better resources. Oh, and the alpha Second Life client has been working OK on Ubuntu x64 with an NVidia 8600 (x64 driver built and installed with Envy)

Some very fun images of the Berkeley 1:4 sims were prepared for the American Geophysical Union Fall 2007 Meeting in San Francisco, under abstract IN13A-0902 on 20071210. The sim hasn’t changed much since then.

With the new year, and a fresh focus on using OpenSim as the server-side vehicle together with Second Life client, I’ve felt that the most effective way to get my point across — of the value that I see in joining immersive 3D simulators to GIS data with the purpose of building 1:1 maps to work inside — could be done better than constant reference to Second Life. So the domain stack grows a bit, and will drop off a bit. Please consider hooking to the stacked domains http://blog.simgis.com or simgis.org as well as the original slgis.org and secondlifegis.com if you’ve got an interest in following these developments.

What’s real-life terrain without real-life imagery to top it off with?

This is a view of Kentfield, Greenbrae, and the Corps of Engineers-enhanced portion of Corte Madera Creek.  The College of Marin athletic fields are on the left, Marin Catholic High School fields in the upper right, and Marin General Hospital in the lower right.  The NAIP imagery is textured on a flat 256 meter square prim fixed at 15 meters scaled elevation, so the sim’s terrain shows above it in several hilly places.

Here, another sim has Phoenix Reservoir and Bill Williams Canyon, both above Ross, California.  Mt. Tamalpais East Peak is behind the lake but does not yet have specific texture on it.  The edge of another 256-meter square prim with the next orthoimage tile to the west is visible in the right.  Two sims farther west, all of Bon Tempe reservoir imagery has been nestled within its proper terrain.

I wish that I could simply use my orthoimagery as a drape over the terrain; alas, that does not seem to be an OpenSim (or Second Life) priority at this time.  None the less, I have already created the first seven sims’ worth of image tiles, and they can be used to provide horizontal control for a great amount of future construction.

Whee! The OpenSim experience grows more wondrous each week. My humble trash-to-testing server, 1 GHz Celeron, 1.5 GB memory, Ubuntu Gutsy Gibbon, is now running at capacity: 49 standalone OpenSim regions, seamlessly covered in 1:4.57-scale real-life terrain, a single user and just the first couple of prims.

The scale works out to a real-life terrain square almost 8.2 kilometers on a side.

The very first couple of prims have gone in to mimic the Gardner Fire Lookout on top of East Peak.

This feels like a bona fide breakthrough for me, knowing that prototypes of Berkeley could be set up, perhaps 49 regions to a test box, on just 10 servers for starters. That would be supporting a 1:1 scale, 500-region model of the City. Much to think about!

My open-source mental nodules have really had a feast these past couple of weeks. First, I had a great insight when I finally listened to a colleague who had spoken well of Ubuntu for some time. When I started using Feisty Fawn and got far, well, and with an attractive and performant result and little effort, I knew that things had evolved well in the past few months.

My previous favorite was a CentOS 4.3 upon which I installed a large commercial mapping package (ESRI ArcIMS 9.1) over Apache and Tomcat. The vendor supported an RHEL configuration and I got close enough for very few problems with CentOS.

But at the home lab, my hardware had to be wrung from random bits of machine that left me with a 1 GHz Celeron / 1.5 GB memory on an ABIT VH6T board, and a new 160 GB PATA disk. First I installed server, but then just relaxed and had Ubuntu Feisty just slide into place. My goal with this box was to dedicate it to OpenSimulator.org testing. In the process of getting Mono, LibSL, and OpenDynamicsEngine all built in place, I also elected to update to Gutsy Gibbon.

Late last week I got first one, then four OpenSim regions working with default starter islands. Next, I applied some spatial resources (ERDAS Imagine) to extract an appropriate amount of real-life digital terrain data. As I am wont to do with my RL-related builds, I scaled down my canyon of interest (Strawberry Canyon above the UC Berkeley campus) to 1/4 scale so it would fit on my four sims.

Just a few tweaks were needed for my extract.
First, ERDAS Imagine has a ton of export formats, and I ended up having success with the IEEE 32-bit floating point format, using swapped bytes (not unswapped, nor any specified -endian flavor).

When I had success, it took me a fiew minutes to realize that the sim was reading in X and Y in a swapped order from the RAW file, and so my terrain had a 90-degree counter-clockwise rotation relative to the original. My original is in WGS84 UTM zone 10 north meters, and has a two-meter posting interval so that it would have come in as a 1/2-scale terrain in OpenSim. I downsampled my terrain clip to a 4-meter interval and rotated it 90 degrees clockwise, then exported a 1K by 1K raw float as described above. It worked just fine and dandy for my first four sims.

Next I worked on a (now I know) naive goal of taking the correponding area’s orothoimagery to drape over my shiny new terrain. I prepared a co-registered 1K by 1K, 24-bit Targa file, then sliced it up into four tiles for my four sims. When I loaded it in using the Estate controls, was I ever surprised! Not only was my pixel-perfect texture tiled all over the place, I realized that there isn’t any way in the Estate controls to change that repeat.

Thanks to nebadon for helping me realize that there’s just no hope for the draped orthoimagery, before I tried too hard to find a magic tweak that wasn’t going to be there. I’ll be able to set up buildings in their proper places using orthoimagery on a sim-sized megaprim, so I’m not totally stumped.

But all in all, my open source nodules are overflowing with wondrous Ubuntu, Mono, and OpenSim progress this past week. I’ve also set myself a goal of modeling Mt. Tamalpais in time to make a poster for the AGU.org Fall Meeting in early December in San Francisco. If I tune the terrain along the fire roads, it might someday be possible to actually ride a virtual mountain bike around the summit of a 1/4-scale East Peak, on an 8 x 8 standalone set of sims.

Nuff for now: here’s Strawberry Canyon towards sunset. Four OpenSim regions with one continuous RL terrain.

The main page at SecondLife.com has lost its counter of resident accounts, but a year’s worth of calibrated experience suggests that the count is rolling over 10 million about now.

Much more concretely, it is with IMMENSE pleasure that we are happy to welcome a new collaborator avatar Rat Dawg, new to Second Life this month, but very persistent with the following of directions from OpenMetaverse. A wonderful recycling of an ancient 1 GHz Celeron with 1.5 GB of memory, fresh installs of Ubuntu Feisty, Mono, and OpenSim have made a new world!

The current SecondLife client 1.18.3 (5) worked swimmingly; sculpties were fine as long as they didn’t emit light (whereupon they reverted to toruses!) and the modifications of terrain were laggy. Otherwise it was an awesome thing to see stuff happening like uploads without any visible debiting of L$, and the uploads were near-instantaneous over the local network.

OpenSim is at Alpha 0.4, but most assuredly worth a look. Having success with it in a Windows-free environment, learning more about Mono and getting major (open source) environmental benefits from it is so fine. Thanks to Ubuntu, I really start to feel like some of my brain cells are awakening from a twelve-year-long, NT-induced stupor. What’s more, I really have a much warmer, fuzzier feeling about collaborating with dotNET developers, knowing that their fine work need not run on servers beholden to Redmond WA.

-=Darb