Jul 07 2017

Unity 3D – Camera Build

One of the beauties, and for some of us the terror, of Unity is its nature as a gaming engine.  Basically it is a sort of 4D Integrated Development Environment (IDE) that makes it relatively efficient to design and build prototype or production game applications—including serious games.

The terror is that Unity is basically an extraordinary IDE for what is most likely C#, and the built-in components are very versatile yet very primitive, which can mean starting from very little.  The beauty includes experiences learning how to create just about anything from those primitives and realizing with some wonder that it’s possible not only to develop, but to actually become productive inside of a week of evenings.  Once I attained my basic understanding of the various (sometimes virtual-literally) moving parts, it was not difficult to identify a feature to add and get that added in one evening, over and over again to where things really accumulate.

Along with the beauty was an occasional burst of joy, as certain coding errors produce psychedelic visual experiences.  It has been very satisfying to reconnect with some old friends, the Quaternions.

https://youtu.be/vjMaYq0J2hQ

No responses yet

Apr 25 2017

Terrain: solved; Platform: Unity;

Published by under SL In General

Following a catch-up read last night of Metaverse Ink from 2011 with many thanks to Prof. Crista Lopes of UCI Informatics for writing it.

Why not?  Unity appears to have a San Francisco SOMA presence, FWIW.  A few hours later, a “Duh” moment.  Why wait so long to go this way?  Why be so intent on synchronous state sharing among distributed clients?  No more questions, just an image to show Unity 5.6.0f3 in action.

Terrain is full-res 1-meter digital surface model, although it has been compressed to unsigned short (u16) integer decimeters with a 10-meter bias to cover creek bathymetry.  Believe me, the effect of negative signed integers on an unsigned int terrain is not pleasant, and the creeks looked like they’d been turned into vertical columns of stratospheric steam.  The swath shown below is 1024 m X 6144 m, has terrain values every square meter, and spans terrain elevations from about 1 m up close to 784 m up high.  At first glance this might look like a green cousin of the obelisk from the movie 2001: A Space Odyssey but in fact it is the fully detailed oblique view of a strip of rendered terrain shown without any atmospheric effects, which makes for some disorienting foreshortening.

I’m still patching terrain 1K-meter squares together manually for a demo.  But this is really looking professional-grade!  Strategically, although this slice of terrain is in Marin county, California, the terrain blocks have been positioned at San Francisco coordinate system of 2013 (WKID/EPSG 7131) so I’m actually using Unity 3D to edit in a GIS grid system.  That opens up the possibility of pouring Vast Tracts of Data into the model from sources like MarinMap.

Unity terrain

Unity 5.6.0f3 Marin dsm1m and MarinMap rgb20cm of 2014

No responses yet

Apr 21 2017

Terrain, yes, and flat orthos

Published by under SL In General

Next reach goal: draping the orthos as terrain textures.

To run better on a 1 vCPU, 1 GB machine with 3GB swap on an EBS SSD drive, the extent was pared back from 8K square to 6K square.  Orthophotos were resampled at 20cm and placed on flat 1km square prims.  Everything aligns with San Francisco county grid (EPSG/WKID 7131.)  This shot is Twin Peaks, view easterly, at sunset.  Look close in the pass and you can see Test User Ruth to verify the scale.  The orthophotos have been placed on both top and bottom of the image prim.

OpenSim 6K model of San Francisco with 20cm orthophotos

Open Simulator 0.9.0-rc2, running on CentOS 7, Amazon t2.micro

No responses yet

Apr 16 2017

Darb Returns to OpenSim at 0.9

Published by under SL In General

The reasons for a return are several, but not that important. The thread that led to this started with a frustration over certain household members’ copious use of the Roblox environment—from a consumer side. The next step was to think “My, but Roblox is little more than a hybrid of Lego/Minecraft figures with a bit more of a Second Life creative side.” After that it was a reactivation of the dormant Darb Dabney account on Second Life itself. Spending some accumulated Linden$ led to a parcel on Jessie, and some of the old muscle memory for builds was reconnected.

Jessie_20170416b

Next, now that the opensimulator.org site can be found again, I realized that there was some new content, a few new faces among the product developers, and a willingness to trade off compatibility with the Linden viewer for an incorporation of larger areas and other useful features—a conscious fork from the Linden path.

So when I finished a load testing project and found myself with 48 more weeks of free-tier Amazon EC2 server, I took a pure CentOS 7 image and went after OpenSim once again. This Easter weekend offered a bit of time to figure out how the tools have all changed, for the better!

I stood back from ArcGIS Pro to avoid distractions and used ArcGIS 10.5 for Desktop to simply design an edit and perform a clip on the stock San Francisco Enterprise GIS Program’s (SFGIS) 50 cm terrain grid, then resampled it to 1 m gridding. The City and County of San Francisco’s new grid system (WKID or EPSG 7131) is ideal as a foundation for OpenSim grids: it’s metric, astronomically aligned,and reasonably small numbers for X and Y coordinates.

I started out getting a 2 km square region going, which was enough to learn what OpenSim needs for local IP (0.0.0.0) and Public IP, where I assigned an Amazon Elastic IP address, then set up DNS to give the server its subdomain here on `3dg.is`. I found that the Linden Second Life Viewer 5.0.3 appears incompatible, but I was able to connect with Singularity Viewer 1.8.7 and see what I needed regarding terrain loads.

Although there are plenty of formats now accepted for terrain maps, many of them are integer-valued and the SFGIS grid is all single precision floating-point so there seems little reason to clobber that precision just to load a grid. Through a variety of tests, I settled on using ArcMap to clip the raster to a square 8192×8192 meter or 6144×6144 meter sample, and saved that in ERDAS .img format. That clip was exported to a GeoTIFF that Adobe Photoshop CC 2017 can read. Mercifully, HDR photography and video has now brought floating-point grayscale into the realm of photographers so I didn’t need to use ERDAS Imagine application. Photoshop made the necessary vertical flip so that OpenSim can read the raster from bottom to top (how backwards!) Then, I opened the flipped no-longer-a-GeoTIFF in a fresh ArcMap document, ignored its lack of georeference, and applied ArcTools > Conversion Tools > From Raster > Raster to Float to produce what Esri calls an .flt file, which is effectively a raw IEEE floating-point array—no header or other information. It is exactly 4xGrid Cells in size. This was renamed to a .r32 for use by OpenSim, and uploaded to the EC2 server using WinSCP.

On the server, I used a binary distribution of opensim-0.9.0.0-rc2.tar.gz and dropped the .r32 in its ./bin directory. The CentOS 7 server was configured to have MySQL (14.14 Distrib 5.7.18 x86_64) and Mono (4.8.0) in order to run the dot net CIL OpenSim.exe and make the server fly.

It was necessary to configure OpenSim to use MySQL rather than the default SQLite, because the extended regions create huge increases in detail and thoroughly exceed limits of SQLite. The 6144×6144 produces an array of 24×24 or 576 Linden regions’ extent. The 8192×8192 upper limit to OpenSim 0.9 forms an array of 32×32 or 1024 Linden regions’ extent.

MySQL defaults had to be expanded at `/etc/my.cnf` and allow big globs of BLOB data, so I added this line to that file, and ran `systemctl stop mysqld; systemctl start mysqld`
“`
max_allowed_packet=500M
“`

To survive the really large regions, the EC2 server needed to be configured for swap, and to make that a bit safer, I added a 2 GB EBS volume housed on SSD, which is the fastest arrangement allowed on this size server.

Notable for this weekend’s efforts was that real San Francisco terrain, with no vertical exaggeration, and 1:1 real-world scale were all used—a first for me, at least with > 500 regions. Here’s a reflection from Twin Peaks, at 1:1 scale, in OpenSim 0.9 running on an Amazon EC2 server at t2.micro (free-tier elegible, 1 vCPU, 1 GB memory, 8 GB disk, 2 GB swap.)

Osim_SF6K_20170416

No responses yet

Jul 23 2016

Esri ArcGIS Pro 1.3

Published by under GIS in general,Vision Statement

For this blog I’m starting a bit of new direction by focusing on Esri tools with available data. Having just completed a draft final version of what will become public-domain building footprints, I’m starting to create 3D City mapping at Level of Detail 1, which I’m inclined to express as a version 1.01 of 3D City.

I’m also in Day 11 of quitting ArcMap cold turkey. Or at least well chilled—with three very narrow returns to ArcGIS 10.4.1 for Desktop for a specific use, then return to Esri ArcGIS Pro 1.3 only. A couple of weeks ago we got the last updates to some building footprints. Now after a couple of day’s work, modeled with bare-earth and first-return LiDAR-derived surfaces, we’ve got extrusions, can run the Procedural textures to give the scene a CityEngine look, and have exported a Level of Detail 1 (call it LoD 1.01) multipatch model.

Gathering statistics from LiDAR was approached in a fairly serious way, with 2 ppsm airborne LiDAR modeled on a 50cm grid with Natural Neighbors (which uses tesselation) for each of [all ground-classified points] and [first-return points]. With some smoothing to attenuate any scanning marks, a [1st return minus bare-earth] difference surface was also calculated and called [height]. The height grid was carefully smoothed with an unsharp mask just enough to eliminate airborne scanner noise, while preserving as much tree canopy detail as possible.

The new building footprints were cleaned and sorted by Shape_Area descending, and assigned an Area_ID in the range [1–177023], then the Area_ID was used with the footprint shapes to define a 25cm zonal grid based on maximum cumulative area that was snapped to the 50cm LiDAR-derived grids. These 25cm cells were used for Zonal Statistics to Table, after converting the three input grids to integer centimeter precision. Zonal statistics on the original floating-point meter grids would not yield either median nor majority values, which were considered useful for stable roof height detection. So all available zonal statistics were run on integer centimeter grids (50cm sampled) of bare-earth [gnd], first-return [1st], and a blurred difference [hgt], sampled over zones of 25cm cells representing each of the 177023 building footprints.

I briefly returned to ArcMap to interactively join the three statistics tables in a more stable way. After that, it was great to get back to Pro 1.3 and edit the schema, renaming and reordering fields, and calculating a few key statistics like minimum ground and median first-return from integer centimeters back into floating-point meters. The minimum ground was used to position each footprint at an absolute elevation based on our bare earth model “tsm50cm” for topographic surface model at 50cm gridding, and the median first-return was used to position the roof at an absolute elevation.

On Day 9 of Pro 1.3, I tried to export the shapes with Procedural textures using Layer 3D to Feature Class, and crashed repeatedly. Even with smaller areas of the city. But finally I got down to just Treasure Island, and it worked with some curious anomalies at edges but with attractive Procedural textures. So I exported to multipatch without color or texture, and got a better-shaped result almost instantly. So, I ran the export for about 1/4 of the city again with Layer 3D to Feature Class, exporting the extruded building footprints to enclosed multipatch 3D boxes—and Pro 1.3 did it in five seconds. Feeling ambitious after that success, I ran the whole city for a set of 177,023 footprints, all Polygon-Z that had been positioned at the lowest NAVD 1988 meters, and the Layer captured the extrusion up to median 1st-return. Not only did that multipatch export complete without crashing, the entire city was done in just 20 seconds.

The Procedural textures were very nice to see. They are oh so precise, but for our city the International Building texture is not so accurate. Still, I was able to tune upper floor height very usefully. And on my little department-issue workstation, I saw all eight CPU threads firing on full while the rendering was taking place. Finding ArcGIS Pro 1.3 running multi-threaded in just the right way to fully utilize the workstation while keeping the interface reasonably responsive—it is a very nice balance indeed. I don’t miss ArcMap at all so far!

No responses yet

Mar 23 2016

Practical application: consensus neighborhoods

Published by under GIS in general

Given: a set of overlapping polygon features in a single Feature Class (FC) and ArcGIS for Desktop

1) Add a field of type Double named “value” to the neighborhood FC and for all features calculate the value “1.0” to this field.

2) Self-Union the neighborhood polygon FC to a new FC named with “_union” appended.
ArcTools > Analysis Tools > Overlay > Union

3) Spatially join the original neighborhood polygon FC to the _union FC, ensuring that each polygon will receive a summary of numeric attributes, and also checking the box Sum to sum those received attributes into a useful value. The desired value should appear in the output FC as attribute “Sum_value”. Finish the output FC name with “_union_join”

neighborhood_sum

4) For visualization, consider converting the _union_join FC to a raster using
ArcTools > Conversion Tools > To Raster > Polygon to Raster

Just choose the Sum_value field as the Value field.

No responses yet

Mar 27 2015

TileStream behind Apache reverse proxy

Published by under SL In General

If you’ve got the issue, then you know why this could matter. If not, then just enjoy consuming Mapbox tiles from somewhere in the world.

Craft a subdomain-accessed reverse proxy with this sort of phrase in Apache httpd.conf on a web server otherwise known as your.host

<VirtualHost   tile.your.host:80>
   ServerName  tile.your.host
   ProxyPass / http://localhost:1111/
   ProxyPassReverse / http://localhost:1111/
</VirtualHost>

Of course, you’ll need to tidy up name resolution for the newly created subnet alias for your server. Once that name resolves, then launch TileStream in some manner like

$ ./index.js --config config.json

With reference to something like this TileStream config.json

{
   "host":  "tile.your.host",
   "tileHost":  "tile.your.host",
   "tilePort":  11111,
   "uiPort":  11111,
   "tiles":  "/your/local/path"
}

For an organization that wishes to share good imagery via TileStream without allowing trivial access to bulk downloads, this can present an issue. TileStream appears rather promiscuous with its download button.

No responses yet

Mar 13 2015

FOSS4G North America 2015 – recovery time

Published by under SL In General

I’m filled to the brim with fresh map ideas from the most amazing crowd that I’ve ever conferenced with. So many ideas were shared with me, many of which change how I view the future of mapping. While there’s still plenty of utility and respect, I sincerely believe that the mapping world has already reached, or is months away from Peak Esri use.

That creates a space for FOSS, and an urgent need to update curricula in the US to keep up. The skills needed are not monolithic desktop apps, but rather freely extensible desktop frameworks—not proprietary servers but those without licensing costs that can be scaled out to cover the world with maps.

No responses yet

Jun 30 2014

SGeoS Esri ArcGIS 10.2.2 for Server Standard Java – Module 1 of 9

Published by under SL In General

Esri ArcGIS 10.2.2 for Server Standard – Java

Build steps for configuration Module-stage-1

  1. Start from completed system Module-stage-0
  2. Create an installation directory for ArcGIS Server
    Name the installation directory with only lowercase letters per the Esri instructions.  Let the installation user own the new directory so that they can perform all necessary actions within.  The example here was chosen to convey version ArcGis Server 10.2.2

    mkdir /ags1022
    chown ags_install /ags1022
    chgrp ags_install /ags1022
    
  3. Enable NFS Export for ArcGIS Server Directory
    Make the installation directory for ArcGIS Server available via NFS.  This will permit Windows 7 Enterprise users (or more likely other ArcGIS Server machines) to connect to it .  Append a line to /etc/exports

    /ags1022  workstationIP(rw,sync,no_root_squash,no_all_squash)
    

    If you find that the check boxes during install seem not to have included NFS as they should: no worries.  It’s like this:

    sudo yum install nfs* -y
    

    Then fire up the share:

    service rpcbind start
    chkconfig rpcbind on
    service nfs start
    chkconfig nfs on
    
  4. Enable SMB for Windows 7 Pro users
    The NFS share is going to be useful among Linux servers, but to develop our services from a Windows desktop, only Windows 7 Enterprise systems have an NFS client built in.  There are open-source NFS clients for Windows, but they are not version-matched with NFS versions most commonly installed on CentOS 6.5.  The main use of NFS is for storage mapping among SGeoS modules on different tiers within a single site, or exchange across SGeoS modules in collaborating environments, such as Dev?Test/QA?Production server transfers.For the Dev machine, we’ll want to enable SMB connections so that any necessary Windows 7 workstation can be configured to connect, particularly Windows 7 Professional machines commonly found deployed through City and County of San Francisco and also at home.SMB can be a less secure means of sharing storage, because it is designed to be compatible with systems that used old and insecure approaches to publishing storage space.  To make this  a clean connection, we’ll configure both iptables as well as mark SELinux to open only the minimum required connection types—but run SELinux in permissive mode to allow SELinux to log but not block actions.Because it is more secure than Workgroup shares, SGeoS modules configure SMB to only work with Active Directory.  Samba Workgroup sharing takes place on other ports that can be left closed.

    It’s easy enough to install the system standard SMB server, but important to configure firewall, give some respect for proper SELinux configuration, and configure the actual SMB shares.
    For Active Directory only we can add these to /etc/sysconfig/iptables
    AGS_01

    Then install

    yum install samba samba-client samba-common ntpd
    

    Verify the installed version; at CentOS 6.5 we get 3.6.9

    smbd --version
    

    Label the served directory to let SELinux know it’s OK to share

    semanage fcontext -a -t public_content_rw_t ‘/ags1022(/.*)?’
    

    Set the Samba services to start at boot time.

    chkconfig smb on
    chkconfig nmb on
    

    With an enterprise install, the standard configuration is found at /etc/samba/smb.conf and should have a section like this to enable a share around ArcGIS Server.  In general, deeper shares with restricted users and allowed client IP are strongly preferred for better server security.

    [ags1022]
       comment = ArcGIS Server 10.2.2 Java
       path = /ags1022
       browseable = yes
       public = no
       writable = yes
       printable = no
    

    It might be desirable to configure Samba to use Active Directory, and according to documentation at http://wiki.samba.org/index.php/Setup_a_Samba_AD_Member_Server it is necessary to have precise time within an AD network, including both a running ntpd and ntp-signd daemons.

    service ntpd stop
    ntpdate wwv.nist.gov
    service ntpd start
    chkconfig ntpdate on
    
  5. Create Esri install and user Linux system accounts
    To create a user account, make it and set its password.
    We need to have a normal user for the ArcGIS Server install, and not install it as root.
    So create a new user and set their password.

    useradd ags_install
    passwd ags_install
    
    useradd ags_user
    passwd ags_user
    
  6. Fulfill Esri-specified system configuration Dependencies
    Work through the Esri-specified dependencies listed http://t.co/f1UoXNrzdr
    yum install Xvfb freetype fontconfig mesa-libGL mesa-libGLUIdentify the hard and soft limits set in the system for file handles and processes

    ulimit -Hn -Hu
    ulimit -Sn -Su
    

    It’s likely default limits are too small to run ArcGIS Server properly, so sudo to edit the file /etc/security/limits.conf  adding these four lines to change settings for
    the ags_intall user
    AGS_02

  7. Enable default httpd
    While it’s possible to install a pre-release Apache 2.4 from RedHat, the default CentOS 6.5 version is 2.2.13—installing more updated versions of web server and OpenSSL are described a couple of sections below.The classic Enterprise approach uses the stock install of httpd 2.2.15 on CentOS 6.5

    yum install httpd
    

    If there’s reason to attach to network (like always), SELinux can be set to allow this

    setsebool -P httpd_can_network_connect on
    

    Poise for open server, but enable only secure browsing with these lines in
    file /etc/sysconfig/iptables for workstations at 10.1.15.x to access via https://
    ags_03

    service httpd restart
    

    If it is desired to have the server always start up the web server, set that to happen

    chkconfig httpd on
    
  8. Install updated httpdOption A:If there’s a desire for an Apache 2.4 httpd on the server, but not the stomach to build one from source, then make the install this way using a software collection  scl  that can install pre-release postings by Red Hat people.  While not a pure enterprise approach, this technique does offer a minimal-risk method to update important framework elements like httpd.
    curl -s http://repos.fedorapeople.org/repos\
       /jkaluza/httpd24/epel-httpd24.repo > /etc/yum.repos.d/epel-httpd24.repo
    
    yum install httpd24-httpd
    

    Then to test it:

    service httpd24-httpd start
       Starting httpd:                                            [  OK  ]
    curl -s http://localhost/ | grep 'Test Page for'
        <title>Test Page for the Apache HTTP Server on Red Hat Enterprise Linux</title>
    


    Option B:
    For security enthusiasts, configure and build from the latest stable Apache source.
    This makes most sense if one also chose to build the very latest OpenSSL from source, in Module-stage0 > Step 7 > Option B. This approach is normal for banking and payment card industries.

    cd /opt/installs
    wget wget http://<some apache mirror site>\
    /apache//apr/apr-1.5.1.tar.gz
    tar xvf apr-1.5.1.tar.gz
    cd apr-1.5.1
    ./configure
    make
    sudo make install
    

    This should place the APR configuration file at /usr/local/apr/bin/apr-1-config

    cd /opt/installs
    wget wget http://<some apache mirror site>\
    /apache//apr/apr-util-1.5.3.tar.gz
    tar xvf apr-util-1.5.3.tar.gz
    cd apr-util-1.5.3
    ./configure --with-apr=/usr/local/apr/bin/apr-1-config
    make
    sudo make install
    

    This should place the APR-util library at /usr/local/apr/lib

    And one more dependency was observed for building httpd:

    yum install  pcre  pcre-devel
    

    Prepare for SSL connections with a self-signed web server certificate

    cd /usr/local
    mkdir pki
    cd pki
    

    Once there, generate a private key for postgresql

    openssl genrsa -out htca.key 8192
    <
    

    Generate a Certificate Signing Request

    openssl req -new -key htca.key -text -out htca.csr
    

    Generate a Self-Signed Key

    openssl x509 -req -days 365 -in htca.csr -signkey htca.key -out htca.crt
    

    Copy these  files to the following locations (DO NOT move them; copy them–then delete)

    cp htca.crt /etc/pki/tls/certs
    cp htca.key /etc/pki/tls/private
    cp htca.csr /etc/pki/tls/private
    chmod 600 /etc/pki/certs/htca.crt /etc/pki/tls/private/htca.*
    rm htca.*
    <
    

    Then we should be ready to actually build an optimized httpd; the  ./configure  is long on options and requires a patch listed here to work with ssl, which it must do.

    cd /opt/installs
    wget http://<some apache mirror>\
    /apache//httpd/httpd-2.4.9.tar.bz2
    tar xvf httpd-2.4.9.tar.bz2
    cd httpd-2.4.9
    export LDFLAGS=”-L/usr/local/lib64”
    
    ./configure  --prefix=/usr/local/httpd \
      --enable-so \
      --enable-pie \
      --with-apr=/usr/local/apr/bin/apr-1-config \
      --enable-ssl \
      --with-ssl=/usr/local/openssl \
      --enable-allowmethods \
      --enable-info \
      --enable-speling \
      --with-mpm=event \
      LDFLAGS=-L/usr/local/lib64 \
      $@
    
    make
    sudo make install
    

    Duplicate some of the enterprise httpd service configuration to make it easier to run the new web server

    su
    cp  /etc/sysconfig/httpd  /etc/sysconfig/httpd2
    cp  /etc/init.d/httpd  /etc/init.d/httpd2
    ln -s  /usr/local/httpd/bin/httpd  /usr/sbin/httpd2
    ln -s  /usr/local/httpd/bin/apachectl  /usr/sbin/apachectl2
    mkdir  /usr/lib64/httpd2
    cp -r /usr/local/httpd/modules /usr/lib64/httpd2
    

    Edit /etc/init.d/httpd2   so that it contains these sort of changes

    apachectl=/usr/sbin/apachectl2
    httpd=${HTTPD-/usr/sbin/httpd2}
    prog=httpd2
    pidfile=${PIDFILE-/var/run/httpd/httpd2.pid}
    lockfile=${LOCKFILE-/var/lock/subsys/httpd2}
    

    Edit /usr/local/httpd/conf/httpd.conf  to redirect all traffic to SSL connections.

    Include conf/extra/httpd-ssl.conf
    Include conf/extra/httpd-mpm.conf
    
    <IfModule unixd_module>
    User apache
    Group apache
    </IfModule>
    
    LoadModule ssl_module modules/mod_ssl.so
    LoadModule socache_shmcb_module modules/mod_socache_shmcb.so
    
    Include conf/extra/httpd-ssl.conf
    
    # Redirect everything to an ssl connection
    # functional Directory is then specified in extra/httpd-ssl.conf
    <VirtualHost *:80>
    ServerName sg11
    Redirect permanent / https://sg11/
    </VirtualHost>
    
    <IfModule dir_module>
    DirectoryIndex  index.html
    </IfModule>
    
    <Files “.ht*”>
    Require all denied
    </Files>
    

    Edit /usr/local/httpd/conf/extra/httpd-ssl.conf  for system content locations and so editors can update content through the SMB share configured at the ArcGIS for Server directory.

    Listen 443
    SSLCipherSuite HIGH:MEDIUM:!aNULL:!MD5
    SSLSessionCache        "shmcb:/usr/local/httpd/logs/ssl_scache(512000)"
    
    <VirtualHost _default_:443>
    
    ServerName sg11:443
    DocumentRoot "/ags1022/html"
    ServerAdmin your.name@here.net
    ErrorLog "/usr/local/httpd/logs/error_log"
    TransferLog "/usr/local/httpd/logs/access_log"
    
    <Location />
    SSLRequire %{SSL_CIPHER_USEKEYSIZE} >= 128
    </Location>
    
    SSLEngine on
    SSLCertificateFile "/etc/your_path_to.crt"
    SSLCertificateKeyFile "/etc/your_path_to_server_private.key"
    
    <FilesMatch "\.(cgi|shtml|phtml|php)$">
    SSLOptions +StdEnvVars
    </FilesMatch>
    <Directory "/usr/local/httpd/cgi-bin">
    SSLOptions +StdEnvVars
    </Directory>
    BrowserMatch "MSIE [2-5]" \
    nokeepalive ssl-unclean-shutdown \
    downgrade-1.0 force-response-1.0
    </VirtualHost>
    

    Back up the site configuration  by making a copy of the modified site configuration files in another location.

    cd /usr/local/httpd
    mkdir /root/httpd_local_conf
    cp -r conf /root/httpd_local_conf
    
  9. Build latest stable Python from source; development server config
    If there’s reason to build PostGIS support framework components with Python support later, it might help to have built the Python locally, so as to appease the linker later.  Reference Python source is from the python.org site. They’ve chosen to compress their archives with a scheme that requires the XZ compression library.  Since Python appears to have a lot of ties to development libraries, it’s been suggested in more than one place to bulk up on some of these tools for smoother builds.
    These may be removed on a production server; they are needed for development.

    yum groupinstall development
    yum install -y zlib-dev openssl-dev sqlite-devel bzip2-devel \
    ncurses-devel readline-devel tk-devel gdbm-devel db4-devel \
    libpcap-devel xz-libs xz-devel
    

    One possible build location is /opt/installs, where a TARFILES directory could be made.
    Create the directory if it doesn’t already exist

    mkdir /opt/installs
    cd !$
    

    Once there, get the compressed source similar to below and decode it

    cd /opt/installs
    wget https://www.python.org/ftp/python/2.7.6/Python-2.7.6.tar.xz
    xz -d Python-2.7.6.tar.xz
    cd ..
    tar xvf TARFILES/Python-2.7.6.tar
    cd Python-2.7.6
    

    Prepare to create a shared library by appending the path /usr/local/lib to /etc/ld.so.conf
    so that it at least looks like:

    include ld.so.conf.d/*.conf
    /usr/local/lib
    

    Then have the linker read the new configuration with

    /sbin/ldconfig
    

    Configure the Python build for alternate location, unicode-32, and shared library. Make it

    ./configure --prefix=/usr/local --enable-shared --with-threads
    make
    

    Let’s not clobber the system’s Python install, and make this the alternate Python install
    This should leave only four minor and/or deprecated bits not found.  Good riddance to them.
    ags_04

    FInally install as an alternate Python so as not to impact any ArcGIS for Server defaults.  Be doubly certain to include the “altinstall” if you’re root.

    make altinstall
    

    Should the make have problems finding libpython2.7.so.1.0,  it could be necessary to create a file /etc/ld.so.conf.d/python2.7.conf   hat lists path /usr/local/python27/lib  if that was chosen as the prefix during config.  After changes there, run this to reload the loader’s configurations

    /sbin/ldconfig
    

    Set up Python build capability by adding Setuptools, then leverage that to install pip and since we’re building the system with Python 2 (and not yet 3), add virtualenv

    mkdir /usr/local/src/Setuptools_py
    cd !$
    wget https://bitbucket.org/pypa/setuptools/raw/bootstrap/ez_setup.py
    python2.7 ez_setup.py
    easy_install-2.7 pip
    pip2.7 install virtualenv
    
  10. Mount the Esri ISO and Prepare for Installing AGS
    (WITH IMPORTANT PRACTICAL NOTE)
    When attaching an ISO image such as Esri installation DVD in the VMware vSphere Client, verify that the ISO has not been mounted in Windows (like to poke around the download) and thus used and locked by Virtual Clone Drive.  If the ISO has been mounted, and one has already tried attaching ISO in vSphere, consider restarting the Windows machine!Oddly, when mounting the ESRI Install DVD ISO,  it appears necessary to launch (or re-launch) the vSphere Client by right-clicking and explicitly using “Run as Administrator”
    With a fresh Windows boot (if needed), and vSphere launched as Administrator, it appears necessary to mount the ESRI ISO with explicit file system type into an existing empty directory such as /cdromThe finesse here seems to be that the login as root and mounting of device can take place in the vSphere console window, then launch a nice large PuTTY ssh window,  log in as ags_install, with home directory in /ags1022, to complete the installationAs root in the console window, after attaching the local ISO, mount the image

    mount -t iso9660 /dev/cdrom /cdrom
    

    Then in the PuTTY window have ags_install verify the mount by looking at all mounted devices; noting the presence of read-only storage at /cdrom

    df
    

    ags_05

    In the PuTTY window, cd into the mounted ISO to see the Setup script.

  11. Install ArcGIS for Server
    Why bother installing a GUI just to run the ArcGIS Server install scripts?  Following the instructions at Esri Resources  the command line interface (CLI) install procedure is most readily described as “Installing ArcGIS for Server silently”  Then, in the cdrom install directory, this wickedly terse statement completely installs all of ArcGIS for Server 10.2.2 into  /ags1022

    su - ags_install
    cd /cdrom/ArcGISServer
    ./Setup -m silent -l Yes /a <path-to-.prvc> /d /ags1022
    

    Fire off the script in silent mode. That’s it. Really.
    If need be, it may be necessary to use the SMB share to copy over the Esri provisioning file to /ags1022, then run the authorizeSoftware script against the .prvc

    /ags1022/arcgis/server/tools/authorizeSoftware -f \
    /ags1022/ArcGISforServerStandardEnterprise_server_xxxxxx.prvc
    

    Then start the post-installation configuration process.
    If server name resolves and ports are open, it’s time to point a browser  at a destination like this and Create New Site

    http://sg11.sfgis.us:6080/arcgis/manager
    

    ags_06

  12. Complete ArcGIS for Server Post-Installation Steps
    This begins with defining an ArcGIS for Server site administrator (not an OS account).
    It’s wise to consider saving this password now in a runbook for the server.
    ags_07Consider keeping the working directories up a bit higher than default location
    ags_08Click Finish, and that’s all that it took.  Seriously easier than it was, once upon a time.
    ags_09
  13. Go Forth and Create Map Services
    Log in, go forth and make many Map and Image services!
    ags_10The new AGS Server Manager console looks more like ArcGIS Online these days:
    ags_11
  14. Secure AGS Manger connections for https-only access
    This will either generate a new cert or provide an opportunity to install an established one.
    Visit not the Manager site, but the Admin one.At first, ArcGIS for Server will be reached by

    http://<server>:6080/arcgis/admin
    

    ags_12

    Go to machines
    ags_13

    In the named machine, Resources: click sslcertificates near the bottom
    ags_14

    To create a new self-signed cert, click generate
    ags_15

    Consider using an Alternative name that is the server’s IP address, to help users who may not have the server name properly resolved in DNS.  That way, only https need be accepted.
    The Subject Alternative Name must be formatted in the style  IP:10.x.x.x
    ags_16

    When the certificate is available, move back up to …/arcgis/admin/machines and go to machine name, and click on Supported Operations:  edit
    ags_17

    Enter the name of the cert that you want to use in Web server SSL Certificate field,
    then click Save Edits.
    ags_18

    After it completes, verify that the chosen cert is displayed.

  15. Enable https-only access for Admin connections to ArcGIS Server
    Starting from  http://<server>:6080/arcgis/admin/security/config
    click on  update then modify the Protocol parameter.  If you haven’t yet verified that the certificate was working and you were able to connect via https:, select the HTTP and HTTPS choice.
    If secure admin connections are working and you were able to connect through
    https://<server>:6443/arcgis/admin/security/config   then it’s OK to select the HTTPS Only choice.
    That’s where you want to end up, but don’t lock yourself out while doing it, so try the two-step approach until verified.  When done, click the Update button.
    ags_19After that, only secured connections to the server will be enabled, at :6443, e.g.

    https://sg11.sfgis.us:6443/arcgis/manager/
    https://sg11:6443/arcgis/manager/
    
  16. Make Publisher or Administrative connection from ArcCatalog In the Catalog tree view, GIS Servers > Add ArcGIS Server > Administer GIS Server  use Server URL in form of
    https://sg11.sfgis.us:6443/arcgis
    

    with Authenication as used in the admin pages above.
    If you’ve used your own self-signed cert, just click through the warning and connect away.
    ags_20

No responses yet

Jun 30 2014

SGeoS Add PostgreSQL 9.2.8 Enterprise Database – Module 2 of 9

Published by under SL In General

PostgreSQL 9.2.8 Enterprise Database Server

Build steps for configuration Module-stage-2

  1. Start from completed system Module-stage-1
    In discussion with Josh Berkus (with Jeff Frost on the line) during PG update meeting of 2014.04.21, our target version is latest in PG 9.2 series, which was 9.2.8 as of that date, with PostGIS 2.0 extension.Per existing EAS data server configurations, data area goes in /data and logs in /pg_xlog
    System prep adapted from instructions in  PostgreSQL Wiki and this posting as well.PostGIS installation with myriad dependencies seemed best documented on this blog post.
  2. Configure YUM repository
    On the CentOS 6.5 system this is /etc/yum.repos.d/CentOS-Base.repo and it’s necessary to add the following lines to avoid having the default RHEL 6.5 version of postgresql installed:
    in [base]

    exclude=postgresql*
    

    in [updates]

    exclude=postgresql*
    
  3. Install PostgreSQL Global Development Group (PGDG) RPM packages for server
    Add these RPMs to replace with packages more current than the CentOS 6.5 default version.Start with installs of libraries upon which the PGDG package depends:

    yum localinstall http://yum.postgresql.org\
       /9.2/redhat/rhel-6-x86_64/postgresql92-libs-9.2.8-1PGDG.rhel6.x86_64.rpm
    

    This solves a dependency that will otherwise cause the subsequent line to be unhappy.
    pg_01

    then go after the PGDG package itself.

    yum localinstall http://yum.postgresql.org\
       /9.2/redhat/rhel-6-x86_64/postgresql92-9.2.8-1PGDG.rhel6.x86_64.rpm
    

    pg_02

    and then the server package

    yum localinstall http://yum.postgresql.org\
       /9.2/redhat/rhel-6-x86_64/postgresql92-server-9.2.8-1PGDG.rhel6.x86_64.rpm
    

    pg_03

    and last, the devel package, required by PostGIS

    yum localinstall  http://yum.postgresql.org\
       /9.2/redhat/rhel-6-x86_64/postgresql92-devel-9.2.8-1PGDG.rhel6.x86_64.rpm
    

    pg_04

  4. Consider adding PGDG contributed package This is a consideration for the development server; probably not needed for production.
    yum localinstall \
    http://yum.postgresql.org/9.2/redhat/rhel-6-x86_64/postgresql92-contrib-9.2.8-1PGDG.rhel6.x86_64.rpm
    

    pg_05

    /* If things just don’t work out right, and a better way forward is found that requires change at an earlier step in the PostgreSQL installation process, it’s OK.  Document what’s going to change next time, and then:

    yum erase postgresql-9.2*
    

    */

  5. Ensure path to installed PG resources is included
    This was necessary for the postgresql user, and in testing for root.
  6. su - postgres
    pwd
    /var/lib/pgsql
    

    There, edit .bash_profile to append these lines:
    pg_06

    And source the edits to make them active

    source .bash_profile
    which pg_ctl
      /usr/pgsql-9.2/bin/pg_ctl
    
  7. Configure major PG locations
    Before initializing PostgreSQL, configure the file system for desired location of data and logs.
    Since the SGeoS machine will sometimes be primarily a database server, choosing root-level locations for data and logs seems merited.Make sure  /etc/sysconfig/pgsql/postgresql.conf exists,  and edit thusly:
    pg_07If these directories will be the locations, then they’d better exist, be owned by PG, grouped with PG, and one should attempt to label an appropriate SElinux context with semanage.

    cd /
    mkdir /data
    mkdir /pg_xlog
    chown postgres /data /pg_xlog
    chgrp postgres /data /pg_xlog
    semanage fcontext -a -t postgresql_db_t “/data(/.*)?”
    semanage fcontext -a -t postgresql_db_t “/pg_xlog(/.*)?”
    su - postgres
    
    mkdir /data/9.2
    mkdir /data/9.2/data
    mkdir /pg_xlog/9.2
    
  8. Initialze PostgreSQL (one time only)
    Carefully verify that your data area is prepared and writeable by postgres user, then initialize. If mistakes are made, consider a cd into /data, then $ rm -Rf 9.2 to try once again.

    initdb -D /data/9.2/data
    

    pg_08

    Continue to tune the data area.  These locations reflect the SFGIS EAS data server style.
    In the interest of SElinux harmony, do the cp, and do not use mv.

    cd /data/9.2/data
    cp  postgresql.conf  postgresql.conf.orig
    cp  pg_hba.conf  pg_hba.conf.orig
    cp -R  pg_xlog  /pg_xlog/9.2
    

    Verify the size of the copy of pg_xlog

    du -s pg_xlog
    

    pg_09

    du -s /pg_xlog/9.2/pg_xlog
    

    pg_10

    Remove the original pg_xlog, and replace with a symbolic link to the copy

    rm -R pg_xlog
    ln -s /pg_xlog/9.2/pg_xlog pg_xlog
    

    This should leave the directory looking like this:

    pwd
    ll
    

    pg_11

  9. Start PostgreSQL and verify it’s running; create test user
    As the postgres user, start the service, from root su – postgres to set environment.(without enviro. variables set)

    pg_ctl start -l /pg_xlog/9.2/pg_xlog/syslog -D /data/9.2/data
    

    (with enviro. variables set)

    pg_ctl start
    

    That should work, so next use psql create a test user and schema to validate connections.
    But first set your postgres db user (db super user) password
    Now is the time to record this assignment in the run book

    psql
    postgres=# ALTER USER Postgres WITH PASSWORD ‘<newpassword>’;
    

    pg_12

  10. Open server (firewall) port to PostgreSQL service
    Exit to root, edit /etc/sysconfig/iptables to open postgresql port with a line similar to this:
    pg_13Then restart iptables to read the new configuration

    service iptables restart
    
  11. Configure PostgreSQL service to accept connections
    As user postgres, edit some postgresql configuration files in /data/9.2/data/

    su - postgres
    cd $PGDATA
    

    (or try the aliased  ‘gopg’)

    Edit postgresql.conf so that listen_address and port are uncommented and set properly for testing purposes.  For production this can be locked down to 127.0.0.1/32  later.
    pg_14

    Edit pg_hba.conf so that it’s simplified to something like this, where the db users are configured to connect locally through loopback (127.0.0.1), which can work through an ssh connection, and for testing also the addresses of Windows workstations from which a GUI administration tool could be run (here 10.x.xx.0/24)
    (using md5 requires that the postgres db user password has been set)
    pg_15

    restart postgresql to get these changes applied

    pg_ctl restart
    

    pg_16

  12. Verify Connections function from Windows workstation
    On the workstation, it’s possible to use a Windows GUI like pgAdmin III to confirm the configuration is working for remote access.  This example describes pgAdmin.
    Launch pgAdmin, and use File > Add Server… to open the New Server Registration dialog.
    Input a reference name for the server in Name, the server’s IP address in Host, and consider testing connection to Maintenance DB postgres with user postgres if you’ve configured things as described above.
    pg_17This should add a line to the Servers object in Server Groups of pgAdmin’s Object Browser.
    pg_18Double-clicking the server object should expand it to show components of the PG instance.
    pg_19
  13. Secure db TCP/IP  Connections  with SSL
    As user postgres, consider testing with a new self-signed cert for use only by PostgreSQL.  The keys can be in an area separated from data.  One approach to do this is to create a directory above the active $PGDATA but still within the installation tree, like /data/9.2/pki

    cd /data/9.2
    mkdir pki
    cd pki
    

    Once there, generate a private key for postgresql

    openssl genrsa -out pgca.key 4096
    

    Generate a Certificate Signing Request

    openssl req -new -key pgca.key -text -out pgca.csr
    

    Generate a Self-Signed Key

    openssl x509 -req -days 365 -in pgca.csr -signkey pgca.key -out pgca.crt
    

    Copy these  files to the following locations (DO NOT move them; copy them–then delete)

    mkdir certs
    mkdir private
    cp pgca.crt /data/9.2/pki/certs
    cp pgca.key /data/9.2/pki/private
    cp pgca.csr /data/9.2/pki/private
    chmod 600 /data/9.2/pki/certs/pgca.crt /data/9.2/pki/private/pgca.*
    rm pgca.*
    

    Once the connections have been verified as working, save a copy of postgresql.conf and proceed to edit the section near Security and Authentication, turning

    ssl = on
    ssl_ciphers = ‘ALL:!ADH:!LOW:!EXP:!MD5:@STRENGTH’
    ssl_cert_file = ‘certs/pgca.crt’
    ssl_key_file = ‘private/pgca.key’
    

    Save edits, then restart PostgreSQL (as postgres user)

    pg_ctl restart
    
  14. On Windows, add PostgreSQL Client libraries to ArcGIS for Desktop
    An SFGIS installer has been prepared for ArcGIS 10.2.2 for Desktop and PostgreSQL 9.2 useConfusingly, even on Windows 7 Pro x86_64 systems, it is essential to load the 32-bit drivers for ArcGIS 10.2 for Desktop—go figure.  Download from http://customers.esri.com the section DBMS Support Files (Client Libraries and Databases).  Avoid any temptation of downloading the Esri PostgreSQL 9.2.2 distribution if you wish to follow the hybrid build.  Instead, expand the PostgreSQL Client Libraries and download PostgreSQL 9.2.2 Client Libraries (Windows) for your workstations.Unpacking those and drilling down will reveal “32bit” and “64bit” folders.  Ignore the 64bit because it is only intended for Windows Server installs of ArcGIS for Server accessing PostgreSQL 9.2.  Instead, use the 32bit folder that is for all versions of ArcGIS for Desktop, even those on 64-bit Windows 7.  There should be six files (you’re in the 64bit folder if there’s only five!)
    Close all running ArcGIS apps, then
    copy all six into    “C:\Program Files (x86)\ArcGIS\Desktop10.2\bin“

    libeay32.dll
    libiconv.dll
    libiconv-2.dll
    libintl.dll
    libpq.dll
    ssleay32.dll
    
  15. Configure PostgreSQL to be enabled at boot if desired
    For administrative convenience it may be desirable to have PostgreSQL always start up at boot.  Here’s how to set that; if the opposite result is desired, substitute “off” for “on”

    chkconfig postgresql-9.2 on
    
  16. Create an SDE database if desired
    For Esri Desktop user convenience it may be desirable to store data in Esri ST_GEOMETRY format as well as PostGIS  PG_GEOMETRY format.  While the PG_ is native to PostGIS, it is necessary to edit the single ArcGIS-enabled database’s SDE schema, sde_dbtune table, GEOMETRY_STORAGE row to have the value PG_GEOMETRY rather than the Esri-default ST_GEOMETRY.
    Of course, PostGIS must be installed before trying to load data with this geometry storage method!

No responses yet

Next »