Darb Dabney’s OpenSim Blog

For ease of QA, there’s nothing quite like shrinking a big multi-region project to get through faster. And to share the joy a bit, this index map is in a public space, on Agni. at Amida 16/12/30

The parcel in Agni (standard Second Life public grid) now has a 1:25 model of Open Berkurodam loaded. There are 159 of the 160 terrain sculpties in place, all with full 1K x 1K ortho image textures. If you find yourself on Agni, stop by to check out the details and see the underside of the terrain sculptie diamonds.

The location is just across the water from original 1:3 scale Berkurodam BART Station. The index map can be found in Amida 16/12/30. Give it a chance to rez, because uncompressed there are 477 MB of Targa image textures represented on 159 terrain sculpties, each of which is specified with a 132×132 bumpmap. In the interest of full disclosure, I have exaggerated the Z dimension here by 50% relative to X and Y, so that the 1:25 scale is horizontal only, and vertical scale is 1:16.6 just to make the terrain more apparent. At this scale, a lot of the immersive experience seems lost and the perspective is quite a bit like Google Earth.

Striving for multiple media channels, I have also uploaded some suitably grainy videos to offer a taste to those who can’t or won’t visit the Agni grid. Believe me, it’s a much sweeter sight at 1600×1200 with the new Windlight viewer, but if one is interested in this sort of rendering, the videos might offer some motivation to explore with the SL client proper.

The longest is 3:39 and starts in Gualala, shows a bit of rezzing of the 1:25 map, does a fairly good job of showing off the texture detail on football fields, then finishes up with a flight over to the Berkurodam BART station 1:3 model, with a glimpse inside the two underground levels.
http://www.youtube.com/watch?v=EjXyBWjGHA8

The shortest video is 0:39 and can be viewed here
http://www.youtube.com/watch?v=IaYd7XD41NY

The next video shows some of both the 1:3 Berkurodam BART station model in Gualala, and the nearby 1:25 OpenBerkurodam index map in an adjacent part of Amida
http://www.youtube.com/watch?v=-AXcDFN6LbA

There is a third one that is still uploading as of this moment

Today I would like to share the inside production notes (it’s quite low tech for the most part) on making terrain sculpties. I have included a full region’s worth of working raw terrain, and a set of four megaprim sculpties that should help to clarify some of my mutterings in earlier posts. Stuff like precisely which values go into the sculptie gradient maps (shown in a spreadsheet), what it looks like when one takes textures that are 960×960 and add a 64-pixel-extent collar around them, and how to actually configure a region to load and display the 250-meter square at Military Grid Reference System / US National Grid 10SEG_6550_9200, with all the necessary bumpmap and texture files, and instructions for both OpenSim console-side and SL client-side actions.

obdam_40h_php This is how I have created the 40 regions using MGRS / US National Grid naming convention, particularly if the terrain has been scaled to 1.024:1 so that exactly 250×250 meters of RL terrain are loaded into each OpenSim region.

obdf_2_7_f32 This is a single region’s raw float terrain file. The original input digital elevation model had been gridded to have postings every 30cm in X and Y. The nominal scale for OpenSim terrain is 1 meter in X and Y. For large grids of real-world regions, there is reason to scale things up slightly so that there are exactly 16 regions per square kilometer. When one does this, as I have here, the samples are 977 mm and the real-world scale is 1.024:1 or a couple of percent larger than life.

sculpt_gradients_132 is the magic for the sculpties, all one needs to do is take the precise spreadsheet values and create three 8-bit grayscale images from them, using a raster program of your choice, to fit 132×132 size for use as a starting point. Then in the middle 130×130 area of the Z-value image (third or blue channel), insert your 8-bit rescaled values of terrain surface. After that, stack the X, Y, and Z grayscales together as Red, Green, and Blue channels to make a single RGB that will be your UV bumpmap.

Working Example single terrain megaprim bumpmap and texture
This is a single sculptie bumpmap+texture set intended to be placed on a megaprim named ‘nw’ that is sized with ‘edit-scale nw 132 132 164′ on the OpenSim console.

Full 1.024:1-scale region with f32 terrain and four megaprim terrain sculpties This is the real deal, one of the 40 regions in the Open Berkurodam sim and I think it’s an interesting part of its steeper area. The link is to a 10MB zip file that is named for the 250-meter square MGRS/US National Grid region that it represents: 10SEG_6550_9200, the grid point at its southwesterly corner. This archive contains a single-float terrain “obdf_2_7.f32″ raw file ready for loading into an OpenSim region; the file name results from a raster dicing script and this is the second region down and seventh region over from the northwesterly corner of the 40-region sim. The archive also contains four pairs of bumpmap+texture Targa files; their file name results from the same dicing script but the indices are higher because these are quarter-region areas.

10SEG_6550_9200_xml Region configuration file for the following example (I neglected to include it above)

To try out the full region set, take an available OpenSim region, and load in the terrain from the OpenSim console with the following sequence

change-region 10SEG_6550_9200
terrain load obdf_2_7.f32
terrain bake

Next, seed the region with four prims. I tend to fly into the middle of the region or teleport and turn left 90 degrees so I am facing northerly, drop four cubes a few meters apart, then name them by their quadrant: ‘nw’, ‘ne’, ’sw’, and ’se’, ensuring that the Prim’s new names have stuck by checking at least one. Then I fly to the outer edge of the region, or just over into the next southerly one. This is not strictly necessary, but it feels like the right thing to do, sort of like walking a safe distance away after having set four large underground charges. That’s because the next step involves super-sizing. In my lab, the OpenSim server is an Ubuntu Linux box running Mono 1.2.6, but my SL client is on Windows XP, and I switch between machines on a KVM switch going from the OpenSim console to SL client, and it is so much easier to see the megaprims if you aren’t inside them after they have been inflated. On the console:

edit-scale nw 132 132 164
edit-scale ne 132 132 164
edit-scale sw 132 132 164
edit-scale se 132 132 164

The Z value is the one I use in the steepest part of the sim, and this makes some really big cubes. I tend to pull them apart far enough to tell them apart.  Be careful here–I’ve hung a sim that was running fine for a week by planting the seed prims not close to the center and then dragging the centroid of the megaprim over into the next region. Sometimes when handling these megaprims it’s handy to ensure that you have the SL client’s draw distance maxed out to 512 meters, and also to zoom the view out from default one notch with “CTL-8″ to trade of field of view with distance from the prims you are handling. So when you’ve got one of the megaprims selected for editing in the SL client, move them into position by keying locations into the Object tab while being very careful not to touch the values in the Size category (thus saving yourself a visit back on the OpenSim console to reinflate the megaprim) When using but four terrain sculptie megaprims per region, their positions in X and Y are always the same, and the Z position will depend on how you’ve rescaled your floating-point terrain to fit into the 8-bit unsigned approximation. For the example megaprims that I have posted, use these:

Prim ‘nw’ XYZ = 64, 192, 188
Prim ‘ne’ XYZ = 192, 192, 188
Prim ’sw’ XYZ = 64, 64, 189.5 (tweaked for amphitheater vs. region terrain)
Prim ’se’ XYZ = 192, 64, 188

Once I can see that the prims have snapped to fully cover the region and are all nearly the same height, I change their Building Block type to Sculpted and see four really large apples that may be somewhat subterranean. If I hadn’t already done so, I use File > Bulk Upload to get all the bumpmap and texture Targa files into inventory. When you have four megaprim sculpties, you should choose the following for Object/Sculpt Texture and Texture/Texture:

Prim ‘nw’ Sculpt = ‘ob40_03_13_z3.tga’ ; Texture = ‘ob40e_03_13.tga’
Prim ‘ne’ Sculpt = ‘ob40_03_14_z3.tga’ ; Texture = ‘ob40e_03_14.tga’
Prim ’sw’ Sculpt = ‘ob40_04_13_z3.tga’ ; Texture = ‘ob40e_04_13.tga’
Prim ’se’ Sculpt = ‘ob40_04_14_z3.tga’ ; Texture = ‘ob403_04_14.tga’

For the sort of appearance that looks best at first, I have kept the background color in the texture to all 255’s and set the Full Bright to checked. That setting does not do well when it’s night in the sim, but it overcomes some sort of fade that is visible in the texture around the edges of the sculptie when Full Bright is not set. More improvements for the future!

Deep thanks to Adam Zaius for pointing out that I hadn’t really made clear these details in the blog. Enjoy!

I’ve gotten into a groove with planting the terrain megaprims, and covered the eastern part of the UC Campus. I’ve also grabbed a video with FRAPS but it’s taking a while to upload to YouTube.

Things I learned tonight: it’s possible to crash OpenSim by dragging megaprims across region boundaries. The warning sign is that the prim appears to lose its name, then all prims in the region lose their names, then a check of the console will show no more OpenSim running!

After a couple of technical issues, I am pleased to offer some machinima views

This is a shot starting at the Greek Theater on the UC Berkeley campus (20080430)
http://www.youtube.com/watch?v=86IVMafq3ik

This is simply how the Open Berkurodam sim looks in its overview map with 40 Regions.
I haven’t refreshed the appearence of the map since loading in the real-world terrain (20080430)
http://www.youtube.com/watch?v=hvGLmtTY0uI

This is a flight eastward over some bare ground, but real-life terrain regions. Flight is in the vicinity of BANCROFT AVE between SHATTUCK AVE and TELEGRAPH AVE (20080430)
http://www.youtube.com/watch?v=56PfQp9viqE

This is a flight into the land of Terrain Megaprim Sculpties.  Of the three scales, this shows the medium and large steepness areas in easterly campus.  At the time this was shot, there were fifteen regions with 60 megaprim sculpties in a contiguous area (20080430)
http://www.youtube.com/watch?v=Q9cElvejrxo

This is a flight from the high point of the sim starting at Lawrence Berkeley National Laboratory, over the Greek Theater, and ending near Wurster Hall at UC Berkeley (20080501)
http://www.youtube.com/watch?v=uBlbB72cpUQ

This is a flight starting near the old Pacific Film Archive building, through an excavation at Underhill Field that was open on 1 July 2006, then up PIEDMONT AVE to GAYLEY AVE past California Memorial Stadium and up to the far NEly corner of the sim in LBNL (20080501)
http://www.youtube.com/watch?v=cealA1QL59s

Enough Videos already!  While you’re at YouTube, check out “OpenSim” as a search term, if you haven’t already!

The manual process I’ve been using to place the terrain megaprims is crying out for automation, but for the moment, I’ve added it only incrementally. I have three OpenSim command scripts now for the three scales of terrain megaprims, and these use edit-scale to supersize the seed prims that I am still putting, four to a reigon, in starter positions by hand.

I’ve had some trouble with my SL client not always showing me all items in inventory. Also, I fixed the one problem terrain megaprim. Apparently it was a naming convention error from when I created the three sets of different-scale terrains as 8-bit, and the problem prim was where I first realized that ERDAS Imagine dicing was starting the tile names anew for each of the rescales, and this was confusing as all three had different upper-left origins.

Even so, I’m only up to having 19 reigons populated with surface terrain sculpties. It is rather satisfying when the megaprims get sized as cubes, positioned appropriately in 3-space, then converted to giant generic sculpties—and when the proper terrain UV is applied, they just snap to the terrain like a fairly decent coat of paint the instant that I select the next prim.

The imagery is tiling very well and that is also satisfying, when I turn on Full Bright and see good detail. It seems that the key to perfect tiling is to adjust the leftmost texture tiles to have a -0.064 horzontal shift to their 1.000 scale texture, and the topmost texture tiles have an 0.064 vertical shift. The upper-leftmost tile has both. I even found a farm in Berkeley–go figure. Looks like they were growing strawberries near Oxford and Hearst on 1 July 2006.

Not to be deterred by having everything going well, however slowly, I opted to test the new distribution of Ubuntu tonight. My “save-xml” scripts ran without warning, but following the OS upgrade, and verification of graphics and browser, my load-xml script only place about one good megaprrim per region. I’ll dup them and get back on track soon. The browser (Swiftweasel) had nary a hiccup, and still runs Flash just fine. My Nvidia display driver kept on working, and I did not need to re-install it as I had been warned when first installing the driver by Envy. The default desktop has a cool Heron on it, the System Monitor has transcended its previous Windows-like aesthetic and improved the dashboard. MySQL seems to be unaffected by the upgrade. Eclipse Europa launches in about 15 seconds. Things feel just a bit faster in many departments, including the rate at which OpenSim draws land.

What’s next:
1) I’d like to finish my way through building the 40-region model on the local OpenSim server here in the lab
2) I’d like to craft a 1:25 scale model of the entire 160-sculptie terrain on Agni in a parcel on Amida not far from to the Berkurodam BART Station 1:3 model.
3) if Hardy Heron holds together for the next few days of testing, I’d like to build a public-facing version of the OpenBerkurodam model, at least as a standalone sim

.

There has been a bit of head scratching as other distractions apparently clouded an obvious scale issue. The first terrain megaprim sculptie project done last month, had available imagery at 30cm.  For that, it only made sense to oversample to 25cm to make 512×512 textures.  That decision led to my adding a collar around the original 512’s until they clicked into the proper size without rescaling on a quarter-region megaprim. With Berkeley, the source imagery is almost 10cm (103mm pixels) and the challenge has been to size the resample so as to make best use of the 1024×1024 texture size limit per prim.

Where I took a wrong turn was trying to proportion the collar that was added to the 512’s, rather than going back to basic principles with sculpties. Bottom line: my efforts of the past week went astray as I allowed confusion to set in, casting about for the proper maximum texture dimensions working down from 1024. (and I’ve got the awkward attempts at 1008, 994, and 978 pixels to prove it).

In fact, the answer is very simple in reference to basic sculptie principles, as the maximum dimensions of the sculptie bumpmap are 32×32, and due to the need to wrap it around to an apex underneath, this can only represent a 30×30 terrain patch. Thus, the maximum imageable area is simply (30/32)*1024, or 960 pixels square, collared out to 1024 square to make each orthophoto tile. This means that an OpenSim 1.024:1 model can accomodate 130mm orthophoto imagery, and I now have 160 tiles ready to go with the bumpmaps.

So far I’ve configured twelve regions with their megaprims, and only one seems to have issues with the height of the sculptie to stay 30cm afloat the terrain surface. Nine of these reigons use the flattest setting, one uses the intermediate, and two use the steepest. Here’s some shots for update’s sake. The full set of orthophoto textures have been uploaded (450 MB of Targa files) and seem to show up reasonably well in inventory. I am using a local MySQL instance on the OpenSim machine for prim storage.

After spending plenty of time getting all the terrain megaprims stamped out, and starting some refinements of how to squeeze the imageable part of the ortho into a portion of the 1024 square texture, I found myself rather unhappy with how lumpy the megaprims were in the flat part of the model. Berkeley has this sort of dual terrain personality (no comment on the residents) that has certain types of details in the distal fan and floodplain parts of town to the westerly, and very different types of details in the hilly and steep areas.

When one loads the natural terrain into OpenSim, the values are 64K of single-precision floating points per region. When loading terrain into a megaprim, there are a mere 900 usable values that must be mashed into an 8-bit integer of Z values. So when the whole sim ranges from 45–267 meters (and it could go up to 581 meters if a sim ran all the way up to Vollmer Peak) , one gets dynamic range issues if all the terrain megaprims are scaled to fit over the entire sim. So to mitigate this, I divided the sim into distal (the “flats”), proximal (the “foots”) , and hills proper (easterly of the Hayward Fault).

The upside of this extra work is better fidelity in the different parts of the model when the floating point values are approximated by 8-bit integers. When I used the entire elevation range for the whole UC Berkeley sim, each integer Z value was 87 cm, or close to three feet. With the sim broken into three regions of megaprim Z-scaling, I have each integer worth 11 cm in the distal, 31 cm in the proximal, and 63 cm in the hills, so everything is a little better everywhere. I’ll fess up to not having the prim placement fully automated, otherwise it would be practical, and perhaps desirable, to use the full dynamic range over just those elevation values in the region (or even in the quadrant of the region that the terrain megaprim covers). But by that point, I’d consider even denser grids of terrain megaprims, and it would then be a different process for representing the terrain.

After all, the whole point of this exercise is to devise a work-around for not being able to load the ortho imagery directly into the region as a draped terrain texture!

Enough blabbing - please enjoy the graphics.

Here is how the terrain maps out versus the buildings

Not intended as digital Cubism, this odd-looking approach makes better terrain megaprims. Really, it does!

In case one knows particular buildings on the UC Berkeley campus or environs, this is how the zoning worked out. There is a certain logic to it, geomorphically.

I’ve been in a bit of a rut the past couple of days, feeling doubt about which way to proceed with configuring the OpenSim side of the UC Berkeley campus 1.024:1 sim. For the first time since I started setting up OpenSim test servers back in October 2007, I was uncertain of my ability to make it work with this project. I rolled back to 0.4, 0.5.0, 0.5.1, and the trunk that worked a couple of days ago would run only 32 regions well, and even at that would stop working, without any use, by morning. All my effort was going into testing out various ways of retreating from the leading edge. In an activity like OpenSim, that’s not a fun place to toil!  Now, after the sim sits quietly through the night, I can teleport from my landing zone in the far SWly region to the far NEly region, and get there pronto.  Plus the 40 Regions are barely consuming 1% of the CPUs.

Realizing that a good 48 hours had passed, one of the things I tried tonight was a fresh grab of the trunk, and that really turned things around for me. With OpenSim 0.5.4_4272 I have the same rocket-fast launch, zippy association of terrain with regions, and I can actually teleport into regions that haven’t resolved their terrain without finding my av hung up. That was all good. Then, I started moving around the ERDAS Imagine data that will be stamped into terrain megaprims, and I was reminded that I’d gone to all the trouble of resampling both terrain and orthoimage for 40 regions, and my diced file naming conventions were already dependent on that entire set of 5 x 8 regions. So rather than fire up the process for making sculptie bump-maps, I went back to the 0.5.4_4272 build, shut it down and went after my region configuration–willing to give it another try at 40 regions. While I was at it I generalized my PHP region XML configurator.

Just for the sake of enjoyment, here are a few views, including nice Windlight night shots. Compared to two days ago, one can notice more of the hill at LBNL, with the fairly intricate grading for service roads and laboratory buildings quite evident in the scene.

Enjoy!

Somehow, things are harder at full scale than they were just a few months ago doing 1:4 work in OpenSim 0.4 for all of Berkeley. The new OpenSim 0.5.4 has some aspects, like initial launch, that are blazingly fast. Still, I struggled with what surely acts like lazy instantiation in the new code. It launches like a rocket, but when I touch it with a client, lots of the same old slogging starts to happen. Very much most significantly for my extensive and largely static work, the current 0.5.4 will not configure stably for more than 35 regions on one (dual-core) server, even with BasicPhysics.

So, after fighting with it for three days, rolling back toward 0.5.0, and finally flopping down to tonight’s SVN trunk, I have reached a spatial compromise that can be shared. I really wrung my heart out trying to pare down the 40-region design posted last week into an “essential” 35 or fewer regions, balancing the virtues of demonstrating 1:1 scale paraverse work in a relevant way to students, Cal faculty, and City government interests. After hours of exploring options, I threw out the northernmost eight regions, to leave a 4×8 array of 32 regions that retain (somewhat selfishly) the Martin Luther King Jr. Civic Center Building, as well as the California Memorial Stadium (for Hayward Fault interests), the Greek Theater, and the Berkeley Community Theater (for musical memories’ sake).

The terrain is of the finest quality for use in OpenSim. I took Alameda County LiDAR-based bare-earth terrain in triangulated irregular network (TIN) form, and using the ArcGIS 9.2 3D Analyst extension, ground out a 30-cm posting interval grid version, in WGS84 UTM 10 north meters, NAVD88 Geoid2003 meters, for processing in Leica Geosystems ERDAS Imagine 9.1 to produce diced tiles that were flipped and exported to raw single-precision floating point, byte-swapped (Motorola style) raw binary terrain tiles that OpenSim so readily digests. To implement my chosen 1.024:1 well-tempered scale, each “1-meter” terrain sample in the sim was actually 976.5 mm in sample interval, simply taking 256 samples over 250 meters of real-world terrain for each sim X- and Y- axis. Thus far, I have neglected to scale the Z in the same way, but when I get to stamping out the terrain megaprims, and need to rescale the terrain for best fit, I’ll rescale.

Here’s two views of the sim running on a dual-core 3.4 GHz server (Ubuntu 7.10 / Mono). One view from the Synchotron building area of Lawrence Berkeley National Lab, at the northeasterly corner of the sim, looking southwesterly.

The other view shows how incredibly detailed and appropriate the LiDAR terrain is when used in 1:1 scale OpenSim application. The view is from near the SEly corner of the sim, and shows the bare-earth expression of the California Memorial stadium, the Greek Theater, and Bancroft Ave running westerly, as visible by its smooth, crowned road grade and gutters, all plainly visible in the terrain.

Enough carefree hours in the main SL Agni grid, already! Back to matters of creation.

Next big thing should be a terrain prototype for civic application. No special business process in mind here, just a demo of the draped imagery on real-life terrain in a way that could scale up city-wide. For starters, there must be a better correspondence between the US National Grid and the dimensions of the simulated region. Sure Neal Stephenson may have suggested binary 2^n dimensionality, and there may be plenty of reasons in the simulator code to make use of the full range of 256 meters. But after more than a handful of regions, the starting corners get downright ugly.

So I won’t do it that way. By scaling up, larger even than real-life, the regions can be built sixteen-to-a square kilometer. In a worldwide sense, except for the matter of 62 or 64 matchlines, the US National Grid (a.k.a. Military Grid Reference System or MGRS) has the whole world in its hands, so harmonizing region design with that grid plan covers a whole lot of ground. To minimize my effort at constructing regions, while planning for worldwide sim grid extensibility, I have chosen to configure the overall sim to represent 250-meter square patches of real earth using each of its 256-meter square regions.

This scales the real-world up a shade in the sim, to (1.024 : 1) but allows every fourth region in X and in Y to start on an exact grid kilometer. That scale produces 16.0000 regions per square kilometer, rather than 15.2588 regions/square km. From the geography side of things, this harmony is attractive since every fourth region will snap to a grid kilometer instead of every 1000th region. Even at that, the grid kilometer that 1000 of those 256 meter regions snap to is 256 kilometers, which is much clumsier to locate by name.

Thus the “well-tempered” moniker for this scale is well deserved, as any real-world USNG/MGRS grid coordinate could then be used to search for the relevant simulator region from a moderately simple bit of string manipulation. For Berkeley, and the western part of California, the zone is “10S” and the 100-kilometer grid within that is “EG” for San Francisco and Berkeley area. Put together, the US National Grid designator for the 100-km square is sometimes called “10SEG”, depending on where folks do or don’t put spaces.

If we always have exactly sixteen (16) regions per square kilometer, then we can use the shorthand version of the USNG grid names that only detail down to 10-meter increments. In this way, a region with its southwesterly corner at WGS84 UTM zone 10 north, 564000 meters Easting, 4191250 meters Northing, can have the US National Grid 10-meter designation of “10SEG 6400 9125″, which could be mashed together without spaces, or used to name a simulator region such as “10SEG_6400_9125″ in a slightly more readable form. For those of us who no longer have youthful eyes, the tiny little display on the Second Life client for the region name motivates the use of spaces.

So here’s a graphic of the plan: a 40-region prototype (5 x 8 regions), which will be configured with only Basic Physics, but real-life LiDAR-based terrain, and four megaprim sculpties per region to drape imagery (10 x 16 terrain sculpties) such as 10cm natural color orthophotography. Here’s where I hope to take this:

Just a note to describe more fun sailing with the new Windlight viewer, and the Wee Tiny Tako 3.2 (that’s a version, not its length. The actual craft is sporting a 2.25-meter waterline!

For my Dad’s birthday this week, I took a sail to the Northern Continent.

Along the way I sailed through ANWR
Finally, I met some unpleasantly private waters that were poorly marked, and my craft reverted to my Inventory, leaving me sitting on a coral reef. Just before this mishap, I enjoyed some fresh winds up to 12 knots northerly and boat speeds up to 7 knots.

Plenty of fun!