Jun 30 2014

SGeoS Esri ArcGIS 10.2.2 for Server Standard Java – Module 1 of 9

Published by under SL In General

Esri ArcGIS 10.2.2 for Server Standard – Java

Build steps for configuration Module-stage-1

  1. Start from completed system Module-stage-0
  2. Create an installation directory for ArcGIS Server
    Name the installation directory with only lowercase letters per the Esri instructions.  Let the installation user own the new directory so that they can perform all necessary actions within.  The example here was chosen to convey version ArcGis Server 10.2.2

    mkdir /ags1022
    chown ags_install /ags1022
    chgrp ags_install /ags1022
    
  3. Enable NFS Export for ArcGIS Server Directory
    Make the installation directory for ArcGIS Server available via NFS.  This will permit Windows 7 Enterprise users (or more likely other ArcGIS Server machines) to connect to it .  Append a line to /etc/exports

    /ags1022  workstationIP(rw,sync,no_root_squash,no_all_squash)
    

    If you find that the check boxes during install seem not to have included NFS as they should: no worries.  It’s like this:

    sudo yum install nfs* -y
    

    Then fire up the share:

    service rpcbind start
    chkconfig rpcbind on
    service nfs start
    chkconfig nfs on
    
  4. Enable SMB for Windows 7 Pro users
    The NFS share is going to be useful among Linux servers, but to develop our services from a Windows desktop, only Windows 7 Enterprise systems have an NFS client built in.  There are open-source NFS clients for Windows, but they are not version-matched with NFS versions most commonly installed on CentOS 6.5.  The main use of NFS is for storage mapping among SGeoS modules on different tiers within a single site, or exchange across SGeoS modules in collaborating environments, such as Dev?Test/QA?Production server transfers.For the Dev machine, we’ll want to enable SMB connections so that any necessary Windows 7 workstation can be configured to connect, particularly Windows 7 Professional machines commonly found deployed through City and County of San Francisco and also at home.SMB can be a less secure means of sharing storage, because it is designed to be compatible with systems that used old and insecure approaches to publishing storage space.  To make this  a clean connection, we’ll configure both iptables as well as mark SELinux to open only the minimum required connection types—but run SELinux in permissive mode to allow SELinux to log but not block actions.Because it is more secure than Workgroup shares, SGeoS modules configure SMB to only work with Active Directory.  Samba Workgroup sharing takes place on other ports that can be left closed.

    It’s easy enough to install the system standard SMB server, but important to configure firewall, give some respect for proper SELinux configuration, and configure the actual SMB shares.
    For Active Directory only we can add these to /etc/sysconfig/iptables
    AGS_01

    Then install

    yum install samba samba-client samba-common ntpd
    

    Verify the installed version; at CentOS 6.5 we get 3.6.9

    smbd --version
    

    Label the served directory to let SELinux know it’s OK to share

    semanage fcontext -a -t public_content_rw_t ‘/ags1022(/.*)?’
    

    Set the Samba services to start at boot time.

    chkconfig smb on
    chkconfig nmb on
    

    With an enterprise install, the standard configuration is found at /etc/samba/smb.conf and should have a section like this to enable a share around ArcGIS Server.  In general, deeper shares with restricted users and allowed client IP are strongly preferred for better server security.

    [ags1022]
       comment = ArcGIS Server 10.2.2 Java
       path = /ags1022
       browseable = yes
       public = no
       writable = yes
       printable = no
    

    It might be desirable to configure Samba to use Active Directory, and according to documentation at http://wiki.samba.org/index.php/Setup_a_Samba_AD_Member_Server it is necessary to have precise time within an AD network, including both a running ntpd and ntp-signd daemons.

    service ntpd stop
    ntpdate wwv.nist.gov
    service ntpd start
    chkconfig ntpdate on
    
  5. Create Esri install and user Linux system accounts
    To create a user account, make it and set its password.
    We need to have a normal user for the ArcGIS Server install, and not install it as root.
    So create a new user and set their password.

    useradd ags_install
    passwd ags_install
    
    useradd ags_user
    passwd ags_user
    
  6. Fulfill Esri-specified system configuration Dependencies
    Work through the Esri-specified dependencies listed http://t.co/f1UoXNrzdr
    yum install Xvfb freetype fontconfig mesa-libGL mesa-libGLUIdentify the hard and soft limits set in the system for file handles and processes

    ulimit -Hn -Hu
    ulimit -Sn -Su
    

    It’s likely default limits are too small to run ArcGIS Server properly, so sudo to edit the file /etc/security/limits.conf  adding these four lines to change settings for
    the ags_intall user
    AGS_02

  7. Enable default httpd
    While it’s possible to install a pre-release Apache 2.4 from RedHat, the default CentOS 6.5 version is 2.2.13—installing more updated versions of web server and OpenSSL are described a couple of sections below.The classic Enterprise approach uses the stock install of httpd 2.2.15 on CentOS 6.5

    yum install httpd
    

    If there’s reason to attach to network (like always), SELinux can be set to allow this

    setsebool -P httpd_can_network_connect on
    

    Poise for open server, but enable only secure browsing with these lines in
    file /etc/sysconfig/iptables for workstations at 10.1.15.x to access via https://
    ags_03

    service httpd restart
    

    If it is desired to have the server always start up the web server, set that to happen

    chkconfig httpd on
    
  8. Install updated httpdOption A:If there’s a desire for an Apache 2.4 httpd on the server, but not the stomach to build one from source, then make the install this way using a software collection  scl  that can install pre-release postings by Red Hat people.  While not a pure enterprise approach, this technique does offer a minimal-risk method to update important framework elements like httpd.
    curl -s http://repos.fedorapeople.org/repos\
       /jkaluza/httpd24/epel-httpd24.repo > /etc/yum.repos.d/epel-httpd24.repo
    
    yum install httpd24-httpd
    

    Then to test it:

    service httpd24-httpd start
       Starting httpd:                                            [  OK  ]
    curl -s http://localhost/ | grep 'Test Page for'
        <title>Test Page for the Apache HTTP Server on Red Hat Enterprise Linux</title>
    


    Option B:
    For security enthusiasts, configure and build from the latest stable Apache source.
    This makes most sense if one also chose to build the very latest OpenSSL from source, in Module-stage0 > Step 7 > Option B. This approach is normal for banking and payment card industries.

    cd /opt/installs
    wget wget http://<some apache mirror site>\
    /apache//apr/apr-1.5.1.tar.gz
    tar xvf apr-1.5.1.tar.gz
    cd apr-1.5.1
    ./configure
    make
    sudo make install
    

    This should place the APR configuration file at /usr/local/apr/bin/apr-1-config

    cd /opt/installs
    wget wget http://<some apache mirror site>\
    /apache//apr/apr-util-1.5.3.tar.gz
    tar xvf apr-util-1.5.3.tar.gz
    cd apr-util-1.5.3
    ./configure --with-apr=/usr/local/apr/bin/apr-1-config
    make
    sudo make install
    

    This should place the APR-util library at /usr/local/apr/lib

    And one more dependency was observed for building httpd:

    yum install  pcre  pcre-devel
    

    Prepare for SSL connections with a self-signed web server certificate

    cd /usr/local
    mkdir pki
    cd pki
    

    Once there, generate a private key for postgresql

    openssl genrsa -out htca.key 8192
    <
    

    Generate a Certificate Signing Request

    openssl req -new -key htca.key -text -out htca.csr
    

    Generate a Self-Signed Key

    openssl x509 -req -days 365 -in htca.csr -signkey htca.key -out htca.crt
    

    Copy these  files to the following locations (DO NOT move them; copy them–then delete)

    cp htca.crt /etc/pki/tls/certs
    cp htca.key /etc/pki/tls/private
    cp htca.csr /etc/pki/tls/private
    chmod 600 /etc/pki/certs/htca.crt /etc/pki/tls/private/htca.*
    rm htca.*
    <
    

    Then we should be ready to actually build an optimized httpd; the  ./configure  is long on options and requires a patch listed here to work with ssl, which it must do.

    cd /opt/installs
    wget http://<some apache mirror>\
    /apache//httpd/httpd-2.4.9.tar.bz2
    tar xvf httpd-2.4.9.tar.bz2
    cd httpd-2.4.9
    export LDFLAGS=”-L/usr/local/lib64”
    
    ./configure  --prefix=/usr/local/httpd \
      --enable-so \
      --enable-pie \
      --with-apr=/usr/local/apr/bin/apr-1-config \
      --enable-ssl \
      --with-ssl=/usr/local/openssl \
      --enable-allowmethods \
      --enable-info \
      --enable-speling \
      --with-mpm=event \
      LDFLAGS=-L/usr/local/lib64 \
      $@
    
    make
    sudo make install
    

    Duplicate some of the enterprise httpd service configuration to make it easier to run the new web server

    su
    cp  /etc/sysconfig/httpd  /etc/sysconfig/httpd2
    cp  /etc/init.d/httpd  /etc/init.d/httpd2
    ln -s  /usr/local/httpd/bin/httpd  /usr/sbin/httpd2
    ln -s  /usr/local/httpd/bin/apachectl  /usr/sbin/apachectl2
    mkdir  /usr/lib64/httpd2
    cp -r /usr/local/httpd/modules /usr/lib64/httpd2
    

    Edit /etc/init.d/httpd2   so that it contains these sort of changes

    apachectl=/usr/sbin/apachectl2
    httpd=${HTTPD-/usr/sbin/httpd2}
    prog=httpd2
    pidfile=${PIDFILE-/var/run/httpd/httpd2.pid}
    lockfile=${LOCKFILE-/var/lock/subsys/httpd2}
    

    Edit /usr/local/httpd/conf/httpd.conf  to redirect all traffic to SSL connections.

    Include conf/extra/httpd-ssl.conf
    Include conf/extra/httpd-mpm.conf
    
    <IfModule unixd_module>
    User apache
    Group apache
    </IfModule>
    
    LoadModule ssl_module modules/mod_ssl.so
    LoadModule socache_shmcb_module modules/mod_socache_shmcb.so
    
    Include conf/extra/httpd-ssl.conf
    
    # Redirect everything to an ssl connection
    # functional Directory is then specified in extra/httpd-ssl.conf
    <VirtualHost *:80>
    ServerName sg11
    Redirect permanent / https://sg11/
    </VirtualHost>
    
    <IfModule dir_module>
    DirectoryIndex  index.html
    </IfModule>
    
    <Files “.ht*”>
    Require all denied
    </Files>
    

    Edit /usr/local/httpd/conf/extra/httpd-ssl.conf  for system content locations and so editors can update content through the SMB share configured at the ArcGIS for Server directory.

    Listen 443
    SSLCipherSuite HIGH:MEDIUM:!aNULL:!MD5
    SSLSessionCache        "shmcb:/usr/local/httpd/logs/ssl_scache(512000)"
    
    <VirtualHost _default_:443>
    
    ServerName sg11:443
    DocumentRoot "/ags1022/html"
    ServerAdmin your.name@here.net
    ErrorLog "/usr/local/httpd/logs/error_log"
    TransferLog "/usr/local/httpd/logs/access_log"
    
    <Location />
    SSLRequire %{SSL_CIPHER_USEKEYSIZE} >= 128
    </Location>
    
    SSLEngine on
    SSLCertificateFile "/etc/your_path_to.crt"
    SSLCertificateKeyFile "/etc/your_path_to_server_private.key"
    
    <FilesMatch "\.(cgi|shtml|phtml|php)$">
    SSLOptions +StdEnvVars
    </FilesMatch>
    <Directory "/usr/local/httpd/cgi-bin">
    SSLOptions +StdEnvVars
    </Directory>
    BrowserMatch "MSIE [2-5]" \
    nokeepalive ssl-unclean-shutdown \
    downgrade-1.0 force-response-1.0
    </VirtualHost>
    

    Back up the site configuration  by making a copy of the modified site configuration files in another location.

    cd /usr/local/httpd
    mkdir /root/httpd_local_conf
    cp -r conf /root/httpd_local_conf
    
  9. Build latest stable Python from source; development server config
    If there’s reason to build PostGIS support framework components with Python support later, it might help to have built the Python locally, so as to appease the linker later.  Reference Python source is from the python.org site. They’ve chosen to compress their archives with a scheme that requires the XZ compression library.  Since Python appears to have a lot of ties to development libraries, it’s been suggested in more than one place to bulk up on some of these tools for smoother builds.
    These may be removed on a production server; they are needed for development.

    yum groupinstall development
    yum install -y zlib-dev openssl-dev sqlite-devel bzip2-devel \
    ncurses-devel readline-devel tk-devel gdbm-devel db4-devel \
    libpcap-devel xz-libs xz-devel
    

    One possible build location is /opt/installs, where a TARFILES directory could be made.
    Create the directory if it doesn’t already exist

    mkdir /opt/installs
    cd !$
    

    Once there, get the compressed source similar to below and decode it

    cd /opt/installs
    wget https://www.python.org/ftp/python/2.7.6/Python-2.7.6.tar.xz
    xz -d Python-2.7.6.tar.xz
    cd ..
    tar xvf TARFILES/Python-2.7.6.tar
    cd Python-2.7.6
    

    Prepare to create a shared library by appending the path /usr/local/lib to /etc/ld.so.conf
    so that it at least looks like:

    include ld.so.conf.d/*.conf
    /usr/local/lib
    

    Then have the linker read the new configuration with

    /sbin/ldconfig
    

    Configure the Python build for alternate location, unicode-32, and shared library. Make it

    ./configure --prefix=/usr/local --enable-shared --with-threads
    make
    

    Let’s not clobber the system’s Python install, and make this the alternate Python install
    This should leave only four minor and/or deprecated bits not found.  Good riddance to them.
    ags_04

    FInally install as an alternate Python so as not to impact any ArcGIS for Server defaults.  Be doubly certain to include the “altinstall” if you’re root.

    make altinstall
    

    Should the make have problems finding libpython2.7.so.1.0,  it could be necessary to create a file /etc/ld.so.conf.d/python2.7.conf   hat lists path /usr/local/python27/lib  if that was chosen as the prefix during config.  After changes there, run this to reload the loader’s configurations

    /sbin/ldconfig
    

    Set up Python build capability by adding Setuptools, then leverage that to install pip and since we’re building the system with Python 2 (and not yet 3), add virtualenv

    mkdir /usr/local/src/Setuptools_py
    cd !$
    wget https://bitbucket.org/pypa/setuptools/raw/bootstrap/ez_setup.py
    python2.7 ez_setup.py
    easy_install-2.7 pip
    pip2.7 install virtualenv
    
  10. Mount the Esri ISO and Prepare for Installing AGS
    (WITH IMPORTANT PRACTICAL NOTE)
    When attaching an ISO image such as Esri installation DVD in the VMware vSphere Client, verify that the ISO has not been mounted in Windows (like to poke around the download) and thus used and locked by Virtual Clone Drive.  If the ISO has been mounted, and one has already tried attaching ISO in vSphere, consider restarting the Windows machine!Oddly, when mounting the ESRI Install DVD ISO,  it appears necessary to launch (or re-launch) the vSphere Client by right-clicking and explicitly using “Run as Administrator”
    With a fresh Windows boot (if needed), and vSphere launched as Administrator, it appears necessary to mount the ESRI ISO with explicit file system type into an existing empty directory such as /cdromThe finesse here seems to be that the login as root and mounting of device can take place in the vSphere console window, then launch a nice large PuTTY ssh window,  log in as ags_install, with home directory in /ags1022, to complete the installationAs root in the console window, after attaching the local ISO, mount the image

    mount -t iso9660 /dev/cdrom /cdrom
    

    Then in the PuTTY window have ags_install verify the mount by looking at all mounted devices; noting the presence of read-only storage at /cdrom

    df
    

    ags_05

    In the PuTTY window, cd into the mounted ISO to see the Setup script.

  11. Install ArcGIS for Server
    Why bother installing a GUI just to run the ArcGIS Server install scripts?  Following the instructions at Esri Resources  the command line interface (CLI) install procedure is most readily described as “Installing ArcGIS for Server silently”  Then, in the cdrom install directory, this wickedly terse statement completely installs all of ArcGIS for Server 10.2.2 into  /ags1022

    su - ags_install
    cd /cdrom/ArcGISServer
    ./Setup -m silent -l Yes /a <path-to-.prvc> /d /ags1022
    

    Fire off the script in silent mode. That’s it. Really.
    If need be, it may be necessary to use the SMB share to copy over the Esri provisioning file to /ags1022, then run the authorizeSoftware script against the .prvc

    /ags1022/arcgis/server/tools/authorizeSoftware -f \
    /ags1022/ArcGISforServerStandardEnterprise_server_xxxxxx.prvc
    

    Then start the post-installation configuration process.
    If server name resolves and ports are open, it’s time to point a browser  at a destination like this and Create New Site

    http://sg11.sfgis.us:6080/arcgis/manager
    

    ags_06

  12. Complete ArcGIS for Server Post-Installation Steps
    This begins with defining an ArcGIS for Server site administrator (not an OS account).
    It’s wise to consider saving this password now in a runbook for the server.
    ags_07Consider keeping the working directories up a bit higher than default location
    ags_08Click Finish, and that’s all that it took.  Seriously easier than it was, once upon a time.
    ags_09
  13. Go Forth and Create Map Services
    Log in, go forth and make many Map and Image services!
    ags_10The new AGS Server Manager console looks more like ArcGIS Online these days:
    ags_11
  14. Secure AGS Manger connections for https-only access
    This will either generate a new cert or provide an opportunity to install an established one.
    Visit not the Manager site, but the Admin one.At first, ArcGIS for Server will be reached by

    http://<server>:6080/arcgis/admin
    

    ags_12

    Go to machines
    ags_13

    In the named machine, Resources: click sslcertificates near the bottom
    ags_14

    To create a new self-signed cert, click generate
    ags_15

    Consider using an Alternative name that is the server’s IP address, to help users who may not have the server name properly resolved in DNS.  That way, only https need be accepted.
    The Subject Alternative Name must be formatted in the style  IP:10.x.x.x
    ags_16

    When the certificate is available, move back up to …/arcgis/admin/machines and go to machine name, and click on Supported Operations:  edit
    ags_17

    Enter the name of the cert that you want to use in Web server SSL Certificate field,
    then click Save Edits.
    ags_18

    After it completes, verify that the chosen cert is displayed.

  15. Enable https-only access for Admin connections to ArcGIS Server
    Starting from  http://<server>:6080/arcgis/admin/security/config
    click on  update then modify the Protocol parameter.  If you haven’t yet verified that the certificate was working and you were able to connect via https:, select the HTTP and HTTPS choice.
    If secure admin connections are working and you were able to connect through
    https://<server>:6443/arcgis/admin/security/config   then it’s OK to select the HTTPS Only choice.
    That’s where you want to end up, but don’t lock yourself out while doing it, so try the two-step approach until verified.  When done, click the Update button.
    ags_19After that, only secured connections to the server will be enabled, at :6443, e.g.

    https://sg11.sfgis.us:6443/arcgis/manager/
    
    https://sg11:6443/arcgis/manager/
    
    
  16. Make Publisher or Administrative connection from ArcCatalog In the Catalog tree view, GIS Servers > Add ArcGIS Server > Administer GIS Server  use Server URL in form of
    https://sg11.sfgis.us:6443/arcgis
    

    with Authenication as used in the admin pages above.
    If you’ve used your own self-signed cert, just click through the warning and connect away.
    ags_20

No responses yet

Jun 30 2014

SGeoS Add PostgreSQL 9.2.8 Enterprise Database – Module 2 of 9

Published by under SL In General

PostgreSQL 9.2.8 Enterprise Database Server

Build steps for configuration Module-stage-2

  1. Start from completed system Module-stage-1
    In discussion with Josh Berkus (with Jeff Frost on the line) during PG update meeting of 2014.04.21, our target version is latest in PG 9.2 series, which was 9.2.8 as of that date, with PostGIS 2.0 extension.Per existing EAS data server configurations, data area goes in /data and logs in /pg_xlog
    System prep adapted from instructions in  PostgreSQL Wiki and this posting as well.PostGIS installation with myriad dependencies seemed best documented on this blog post.
  2. Configure YUM repository
    On the CentOS 6.5 system this is /etc/yum.repos.d/CentOS-Base.repo and it’s necessary to add the following lines to avoid having the default RHEL 6.5 version of postgresql installed:
    in [base]

    exclude=postgresql*
    

    in [updates]

    exclude=postgresql*
    
  3. Install PostgreSQL Global Development Group (PGDG) RPM packages for server
    Add these RPMs to replace with packages more current than the CentOS 6.5 default version.Start with installs of libraries upon which the PGDG package depends:

    yum localinstall http://yum.postgresql.org\
       /9.2/redhat/rhel-6-x86_64/postgresql92-libs-9.2.8-1PGDG.rhel6.x86_64.rpm
    

    This solves a dependency that will otherwise cause the subsequent line to be unhappy.
    pg_01

    then go after the PGDG package itself.

    yum localinstall http://yum.postgresql.org\
       /9.2/redhat/rhel-6-x86_64/postgresql92-9.2.8-1PGDG.rhel6.x86_64.rpm
    

    pg_02

    and then the server package

    yum localinstall http://yum.postgresql.org\
       /9.2/redhat/rhel-6-x86_64/postgresql92-server-9.2.8-1PGDG.rhel6.x86_64.rpm
    

    pg_03

    and last, the devel package, required by PostGIS

    yum localinstall  http://yum.postgresql.org\
       /9.2/redhat/rhel-6-x86_64/postgresql92-devel-9.2.8-1PGDG.rhel6.x86_64.rpm
    

    pg_04

  4. Consider adding PGDG contributed package This is a consideration for the development server; probably not needed for production.
    yum localinstall \
    
    http://yum.postgresql.org/9.2/redhat/rhel-6-x86_64/postgresql92-contrib-9.2.8-1PGDG.rhel6.x86_64.rpm
    
    

    pg_05

    /* If things just don’t work out right, and a better way forward is found that requires change at an earlier step in the PostgreSQL installation process, it’s OK.  Document what’s going to change next time, and then:

    yum erase postgresql-9.2*
    

    */

  5. Ensure path to installed PG resources is included
    This was necessary for the postgresql user, and in testing for root.
  6. su - postgres
    pwd
    /var/lib/pgsql
    

    There, edit .bash_profile to append these lines:
    pg_06

    And source the edits to make them active

    source .bash_profile
    which pg_ctl
      /usr/pgsql-9.2/bin/pg_ctl
    
  7. Configure major PG locations
    Before initializing PostgreSQL, configure the file system for desired location of data and logs.
    Since the SGeoS machine will sometimes be primarily a database server, choosing root-level locations for data and logs seems merited.Make sure  /etc/sysconfig/pgsql/postgresql.conf exists,  and edit thusly:
    pg_07If these directories will be the locations, then they’d better exist, be owned by PG, grouped with PG, and one should attempt to label an appropriate SElinux context with semanage.

    cd /
    mkdir /data
    mkdir /pg_xlog
    chown postgres /data /pg_xlog
    chgrp postgres /data /pg_xlog
    semanage fcontext -a -t postgresql_db_t “/data(/.*)?”
    semanage fcontext -a -t postgresql_db_t “/pg_xlog(/.*)?”
    su - postgres
    
    mkdir /data/9.2
    mkdir /data/9.2/data
    mkdir /pg_xlog/9.2
    
  8. Initialze PostgreSQL (one time only)
    Carefully verify that your data area is prepared and writeable by postgres user, then initialize. If mistakes are made, consider a cd into /data, then $ rm -Rf 9.2 to try once again.

    initdb -D /data/9.2/data
    

    pg_08

    Continue to tune the data area.  These locations reflect the SFGIS EAS data server style.
    In the interest of SElinux harmony, do the cp, and do not use mv.

    cd /data/9.2/data
    cp  postgresql.conf  postgresql.conf.orig
    cp  pg_hba.conf  pg_hba.conf.orig
    cp -R  pg_xlog  /pg_xlog/9.2
    

    Verify the size of the copy of pg_xlog

    du -s pg_xlog
    

    pg_09

    du -s /pg_xlog/9.2/pg_xlog
    

    pg_10

    Remove the original pg_xlog, and replace with a symbolic link to the copy

    rm -R pg_xlog
    ln -s /pg_xlog/9.2/pg_xlog pg_xlog
    

    This should leave the directory looking like this:

    pwd
    ll
    

    pg_11

  9. Start PostgreSQL and verify it’s running; create test user
    As the postgres user, start the service, from root su – postgres to set environment.(without enviro. variables set)

    pg_ctl start -l /pg_xlog/9.2/pg_xlog/syslog -D /data/9.2/data
    

    (with enviro. variables set)

    pg_ctl start
    

    That should work, so next use psql create a test user and schema to validate connections.
    But first set your postgres db user (db super user) password
    Now is the time to record this assignment in the run book

    psql
    postgres=# ALTER USER Postgres WITH PASSWORD ‘<newpassword>’;
    

    pg_12

  10. Open server (firewall) port to PostgreSQL service
    Exit to root, edit /etc/sysconfig/iptables to open postgresql port with a line similar to this:
    pg_13Then restart iptables to read the new configuration

    service iptables restart
    
  11. Configure PostgreSQL service to accept connections
    As user postgres, edit some postgresql configuration files in /data/9.2/data/

    su - postgres
    cd $PGDATA
    

    (or try the aliased  ‘gopg’)

    Edit postgresql.conf so that listen_address and port are uncommented and set properly for testing purposes.  For production this can be locked down to 127.0.0.1/32  later.
    pg_14

    Edit pg_hba.conf so that it’s simplified to something like this, where the db users are configured to connect locally through loopback (127.0.0.1), which can work through an ssh connection, and for testing also the addresses of Windows workstations from which a GUI administration tool could be run (here 10.x.xx.0/24)
    (using md5 requires that the postgres db user password has been set)
    pg_15

    restart postgresql to get these changes applied

    pg_ctl restart
    

    pg_16

  12. Verify Connections function from Windows workstation
    On the workstation, it’s possible to use a Windows GUI like pgAdmin III to confirm the configuration is working for remote access.  This example describes pgAdmin.
    Launch pgAdmin, and use File > Add Server… to open the New Server Registration dialog.
    Input a reference name for the server in Name, the server’s IP address in Host, and consider testing connection to Maintenance DB postgres with user postgres if you’ve configured things as described above.
    pg_17This should add a line to the Servers object in Server Groups of pgAdmin’s Object Browser.
    pg_18Double-clicking the server object should expand it to show components of the PG instance.
    pg_19
  13. Secure db TCP/IP  Connections  with SSL
    As user postgres, consider testing with a new self-signed cert for use only by PostgreSQL.  The keys can be in an area separated from data.  One approach to do this is to create a directory above the active $PGDATA but still within the installation tree, like /data/9.2/pki

    cd /data/9.2
    mkdir pki
    cd pki
    

    Once there, generate a private key for postgresql

    openssl genrsa -out pgca.key 4096
    

    Generate a Certificate Signing Request

    openssl req -new -key pgca.key -text -out pgca.csr
    

    Generate a Self-Signed Key

    openssl x509 -req -days 365 -in pgca.csr -signkey pgca.key -out pgca.crt
    

    Copy these  files to the following locations (DO NOT move them; copy them–then delete)

    mkdir certs
    mkdir private
    cp pgca.crt /data/9.2/pki/certs
    cp pgca.key /data/9.2/pki/private
    cp pgca.csr /data/9.2/pki/private
    chmod 600 /data/9.2/pki/certs/pgca.crt /data/9.2/pki/private/pgca.*
    rm pgca.*
    

    Once the connections have been verified as working, save a copy of postgresql.conf and proceed to edit the section near Security and Authentication, turning

    ssl = on
    ssl_ciphers = ‘ALL:!ADH:!LOW:!EXP:!MD5:@STRENGTH’
    ssl_cert_file = ‘certs/pgca.crt’
    ssl_key_file = ‘private/pgca.key’
    

    Save edits, then restart PostgreSQL (as postgres user)

    pg_ctl restart
    
  14. On Windows, add PostgreSQL Client libraries to ArcGIS for Desktop
    An SFGIS installer has been prepared for ArcGIS 10.2.2 for Desktop and PostgreSQL 9.2 useConfusingly, even on Windows 7 Pro x86_64 systems, it is essential to load the 32-bit drivers for ArcGIS 10.2 for Desktop—go figure.  Download from http://customers.esri.com the section DBMS Support Files (Client Libraries and Databases).  Avoid any temptation of downloading the Esri PostgreSQL 9.2.2 distribution if you wish to follow the hybrid build.  Instead, expand the PostgreSQL Client Libraries and download PostgreSQL 9.2.2 Client Libraries (Windows) for your workstations.Unpacking those and drilling down will reveal “32bit” and “64bit” folders.  Ignore the 64bit because it is only intended for Windows Server installs of ArcGIS for Server accessing PostgreSQL 9.2.  Instead, use the 32bit folder that is for all versions of ArcGIS for Desktop, even those on 64-bit Windows 7.  There should be six files (you’re in the 64bit folder if there’s only five!)
    Close all running ArcGIS apps, then
    copy all six into    “C:\Program Files (x86)\ArcGIS\Desktop10.2\bin“

    libeay32.dll
    libiconv.dll
    libiconv-2.dll
    libintl.dll
    libpq.dll
    ssleay32.dll
    
  15. Configure PostgreSQL to be enabled at boot if desired
    For administrative convenience it may be desirable to have PostgreSQL always start up at boot.  Here’s how to set that; if the opposite result is desired, substitute “off” for “on”

    chkconfig postgresql-9.2 on
    
  16. Create an SDE database if desired
    For Esri Desktop user convenience it may be desirable to store data in Esri ST_GEOMETRY format as well as PostGIS  PG_GEOMETRY format.  While the PG_ is native to PostGIS, it is necessary to edit the single ArcGIS-enabled database’s SDE schema, sde_dbtune table, GEOMETRY_STORAGE row to have the value PG_GEOMETRY rather than the Esri-default ST_GEOMETRY.
    Of course, PostGIS must be installed before trying to load data with this geometry storage method!

No responses yet

Jun 30 2014

SGeoS Build PostGIS 2.1.3 with SFCGAL and GTK+ – Module 3 of 9

Published by under SL In General

PostGIS 2.1.3 with SFCGAL geometry and GTK+ annotation

Build steps for configuration Module-stage-3
PostGIS is a very powerful and highly interdependent Open Source GIS tool.
Despite its light and attractive project page and logo,
PostGIS_cute_logo
numerous PostGIS dependencies, some of which have themselves numerous dependencies,  can turn a source build process into a wrestle with a snarling beast.
PostGIS_huge_elephant_thing

The following install procedure began by adapting a particularly insightful and succinct blog post and continued to ensure that all available PostGIS 2.1 extensions, and its most precise geometric capabilities were enabled.

A wise choice might be to use the PGDG package rather than building this all from source. http://yum.postgresql.org/9.2/redhat/rhel-6-x86_64/

Open Source is a capable and powerful approach, yet each contributor’s work becomes most efficient when only filling gaps between existing tools.  As a result, a high-level project such as PostGIS is built with many dependencies on the components it has connected; these dependencies may themselves have several levels of dependencies beneath that provide the capabilities used in that project.

Every one of these gets and clones is used in subsequent parts of this most-involved step, where the extended functions of PostGIS 2.1 have been enabled.  Most steps are built from source to incorporate the latest stable build.  As with all unix command lines, each and every character should be viewed as important. Places in this document where that’s the case  have been shown in Courier font.

  1. Start from completed system Module-stage-2
  2. Option A: Install PostGIS from package  To just get it done and move on to the next stage; this is a smart choice for most circumstances, unless one is intent on the very latest PostGIS / CGAL features.
    wget http://yum.postgresql.org/9.2/ \
    redhat/rhel-6-x86_64/postgis2_92-2.1.3-1.rhel6.x86_64.rpm
    yum install postgis2_92-2.1.3-1.rhel6.x86_64.rpm
    
  3. Option B (steps 4–31) build PostGIS from source with latest stable versions of dependencies:
  4. Obtain PostGIS and extensions sources used in subsequent steps for the first SGeoS testbed server.  This approach will help clarify the many components upon which PostGIS depends. Consider visiting the site in the leading parts of each URL then making an informed choice about which versions to download and build from source when starting fresh.
    mkdir /opt/installs
    cd !$
    wget http://download.osgeo.org/postgis/source/postgis-2.1.3.tar.gz
    wget http://download.osgeo.org/proj/proj-4.8.0.tar.gz
    wget http://download.osgeo.org/geos/geos-3.4.2.tar.bz2
    git clone https://github.com/json-c/json-c.git
    wget https://s3.amazonaws.com/json-c_releases/releases/json-c-0.12-nodoc.tar.gz
    wget http://www.hdfgroup.org/ftp/HDF5/current/src/hdf5-1.8.13.tar.gz
    wget http://download.osgeo.org/gdal/1.11.0/gdal-1.11.0.tar.gz
    wget http://sourceforge.net\
    /projects/cunit/files/CUnit/2.1-2/CUnit-2.1-2-src.tar.bz2
    wget http://sourceforge.net\
    /projects/freetype/files/ftjam/2.5.2/ftjam-2.5.2.tar.bz2
    wget http://sourceforge.net\
    /projects/dblatex/files/latest/download?source=files
    wget http://www.cmake.org/files/v2.8/cmake-2.8.12.2.tar.gz
    wget http://sourceforge.net\
    /projects/boost/files/boost/1.55.0/boost_1_55_0.tar.bz2
    wget https://ftp.gnu.org/gnu/gmp/gmp-6.0.0a.tar.bz2
    wget http://www.mpfr.org/mpfr-current/mpfr-3.1.2.tar.bz2
    wget http://download.qt-project.org\
    /official_releases/qt/4.8/4.8.6/qt-everywhere-opensource-src-4.8.6.tar.gz
    wget http://www.algorithmic-solutions.info\
    /free/Packages/LEDA-6.3-free-fedora-core-8-64-g++-4.1.2-mt.tar.gz
    wget https://gforge.inria.fr/frs/download.php/file/33524/CGAL-4.4.tar.bz2
    wget https://github.com/Oslandia/SFCGAL/archive/v1.0.4.tar.gz -O sfcgal-1.0.4.tar.gz
    wget http://www.us.apache.org/dist/ant/binaries/apache-ant-1.9.4-bin.tar.bz2
    wget http://pkgconfig.freedesktop.org/releases/pkgconfig-0.18.tar.gz
    wget ftp://sourceware.org/pub/libffi/libffi-3.1.tar.gz
    wget http://ftp.acc.umu.se/pub/gnome/sources/glib/2.41/glib-2.41.0.tar.xz
    wget http://ftp.gnome.org/pub/gnome/sources/gtk+/3.12/gtk+-3.12.2.tar.xz
    wget http://ftp.stack.nl/pub/users/dimitri/doxygen-1.8.7.src.tar.gz
    
  5. Configure a few dependencies  Using these prepared packages saves system build time
    yum -y install tetex-tex4ht
    yum -y install libxml2-devel
    yum -y install ImageMagick*
    yum -y install gcc-c++  json-c-devel.x86_64
    yum -y install java-1.7.0-openjdk  java-1.7.0-openjdk-devel
    yum -y install java-devel xerces-j2
    yum -y install mesa-libGL mesa-libGL-devel mesa-libGLU-devel
    

    Notes regarding dependencies for building PostGIS with all extensions and documentation:
    - The teTeX package implemented TeX document typesetting for unix-like systems, the tex4ht package is the TeX for Hypertext, converting typeset technical documents into HTML and XML.
    - The development packages for XML is a C library for eXtensibe Markup Language (XML) that is part of the GNOME (“genome”) project that builds desktop frameworks for Linux.  The libxml2 package is also used outside of the desktop environment to handle XML interchange in C programming.
    - The ImageMagick package provide some file format conversion and image processing functions that can be used within several programming environments
    - The GNU C++ compiler is a vast package to build, and the default version works for most source, and the JavaScript Object Notation development bundle is sought for by PostGIS
    - Open JDK is an Open Source implementation of Java Standard Edition from Sun Microsystems prior to its acquisition by Oracle.  Open JDK is the reference implementation of the Java Development Kit.
    - Java development tools are augmented by the Xerces2 Java Parser to implement XML schema
    - mesa-libGL are OpenGL development packages installed from the MIT mesa implementation.
    The main Mesa package was installed for ArcGIS, so expect only the devel packages to be added.
    - Doxygen is a documentation generator used to build PostGIS documentation

  6. Install proj-4 from source, the -j2 flag allows compilation in two threads for a 1-core machine. The PROJ.4 package performs all manner of geographic projections and transformations and is used by PostGIS.
    cd /opt/installs
    tar xvf proj-4.8.0.tar.gz
    cd proj-4.8.0
    ./configure
    make -j2
    make check
    make install
    ldconfig
    ln -s /usr/local/lib/pkgconfig/proj.pc /usr/lib64/pkgconfig/
    
  7. Install GEOS from source,  this requires the c++ compiler if it’s not already installed. This is the Geometry Engine – Open Source (GEOS), ported from Java Topology Suite to C++, and is the PostGIS default for operations not performed by CGAL.
    cd /opt/installs
    tar xvf geos-3.4.2.tar.bz2
    cd geos-3.4.2
    make -j2
    make install
    ldconfig
    
  8. Install json-c from source,  on CentOS 6.5 this required an updated autogen as of 2014.05.19. The JavaScript Object Notation enables attribute-value pair object communication. JSON-C provides this capability for C language programming purposes and is used by PostGIS.
    cd /opt/installs/
    tar xvf json-c-0.12-nodoc.tar.gz
    cd json-c-0.12
    ./autogen.sh
    ./configure
    autoreconf -fvi
    make -j2
    make install
    ldconfig
    ln -s /usr/local/lib/pkgconfig/json.pc /ur/lib64/pkgconfig
    
  9. Install HDF5 from source,  Used by GDAL, takes over eight minutes to make a default config; the make check provides some peace of mind given the vast number of warnings thrown.  This is the Hierarchical Data Format developed by NCSA in the US, and used by Python, Matlab, and Java.
    cd /opt/installs/
    tar xvf hdf5-1.8.13.tar.gz
    cd hdf5-1.8.13
    ./configure
    make
    make check
    make install
    
  10. Install GDAL from source,  use latest Python 2.7.6 by running in virtualenv for this; be patient. This is the Geospatial Data Abstraction Library (GDAL) to transform vector and raster data formats through a common GDAL abstract raster type and a common OGR abstract vector data type.  With these OGR types merged, GDAL functions like an Open Source version of Safe Software FME.
    cd /opt/installs
    virtualenv venv
    source venv/bin/activate
    tar xvf gdal-1.11.0.tar.gz
    cd gdal-1.11.0
    ./configure --with-python
    make -j2
    make install
    ldconfig
    
  11. Install CUnit from source,  supports unit testing; version 2.1-2 installs with configure, so use that version.  CUnit provides a testing framework that can be used by C programmers.  This module is used in PostGIS to construct standard test suites to verify function of code after it has been built.
    cd /opt/installs
    tar xvf CUnit-2.1-2-src.tar.bz2
    cd CUnit-2.1-2
    ./configure
    make
    make install
    ldconfig
    ln -s /usr/local/lib/pkgconfig/cunit.pc /usr/lib64/pkgconfig
    
  12. Install dblatex from egg,  depends on earlier  easy_install  Python package when building new Python from source (Module-0, Step 8)  and a manual install of DocBook DTD and an initial catalog for PostGIS documentation.  The first computer-based typesetting system TeX was extended with macro tags to become LaTeX, and a specialized set of macros for technical documentation were applied to create DocBook LaTeX, or dblatex.  With it, one writes documentation once in a neutral format, then exports it to many different presentations.
    cd /usr/local/share
    mkdir xml  xml/docbook  xml/docbook/dtd  xml/docbook/dtd/5.0
    cd !$
    wget -O docbook5.dtd http://docbook.org/xml/5.0/dtd/docbook.dtd
    cd /usr/local/share/xml
    xmlcatalog --noout --create docbcatalog
    xmlcatalog -noout --add ‘public’ ‘-//OASIS//DTD DocBook XML V5.0//EN’ \
    ‘file://usr/local/share/xml/docbook/dtd/5.0/docbook5.dtd’  docbcatalog
    
    http://vault.centos.org/6.5/os/Source/SPackages/docbook5-style-xsl-1.75.2-4.el6.src.rpm
    
    # install DocBook style sheets
    yum install docbook5-style*
    cd /usr/share/sgml/docbook
    ln -s xsl-ns-stylesheets-1.75.2 xsl-stylesheets
    easy_install dblatex
    
  13. Install CMake from source,  yet another build tool, this one used by CGAL.  The CGAL package is vast and written in C++, so a more powerful make tool was used than the C-oriented system defaults.
    cd /opt/installs
    tar xvf cmake-2.8.12.2.tar.gz
    cd cmake-2.8.12.2
    ./bootstrap
    make
    make install
    
  14. Install Boost from source,  a vast collection of C++ extensions used by CGAL; be patient as it can take 12 minutes to compile.  Boost libraries provide standard tools that underlie much of the numerical programming in CGAL, boosting the productivity of the programmers it serves.
    cd /opt/installs
    tar xvf boost_1_55_0.tar.bz2
    cd boost_1_55_0
    ./bootstrap.sh
    ./b2
    ./b2 install --prefix=/usr/local
    ldconfig
    
  15. Install GMP from source,  This is the arithmetic library used by MPFR.  The recursive name GNU’s Not Unix (GNU) brands the original Open Source porting of Unix.  The GNU Multiple Precision  (GMP) arithmetic library overcomes common floating-point limits with extensible precision for calculations.
    cd /opt/installs
    tar xvf gmp-6.0.0a.tar.bz2
    cd gmp-6.0.0
    make
    make check
    make install
    ldconfig
    
  16. Install MPFR from source,  multi-precision floating point library used by CGAL.  This is the GNU Multiple Precision Floating-point Reliably (MPFR) library, where you can divide by zero and not crash.
    cd /opt/installs
    tar xvf mpfr-3.1.2.tar.bz2
    cd mpfr-3.1.2
    ./configure
    make
    make check
    make install
    
  17. Install Qt4 from source,  C++ programming framework used by CGAL; a huge build that could take 45 minutes or more to compile. The Qt (“cutie”) framework was developed through a company Quasar Technologies and provides a library of interface graphic widgets for designers across many platforms.  When Nokia bought Quasar, the name became Qt.
    NOTE: ArcGIS mobile applications now use ArcGIS Runtime SDK for Qt, so this package can have use for both Esri and CGAL purposes.

    cd /opt/installs
    tar xvf qt-everywhere-opensource-src-4.8.6.tar.gz
    mv  qt-everywhere-opensource-src-4.8.6  qt-4.8.6
    export QTDIR=/opt/installs/qt-4.8.6
    cd qt-4.8.6
    ./configure
    o
    yes
    gmake
    gmake install
    
  18. Install LEDA object libraries,  graph and network system used by CGAL.  The Library of Efficient Data Types and Algorithms (LEDA) is from Max Planck Institute provides computational geometry and graph theory algorithms.  Its distrubutor Algorithmic Solutions Software GmbH licenses source code for commercial use, and offers the library of binary functions for free.  We use the binaries.
    cd /opt/installs
    tar xvf LEDA-6.3-free-fedora-core-8-64-g++-4.1.2-mt.tar
    mv  LEDA-6.3-free-fedora-core-8.64-g++-4.1.2-mt   LEDA-6.3
    export LEDAROOT=/opt/installs/LEDA-6.3
    
  19.  Install CGAL from source,  general-purpose spatial math processing from INRIA France wrappered by SFCGAL.  The Computational Geometry Algorithms Library (CGAL) is a C++ library for efficient and reliable geometric algorithms.  It is available for use with Open Source software for free and is licensed for commercial use.  Because it incorporates arbitrary precision and floating-point reliability, it might produce more accurate and reliable spatial queries.  For this, it was considered worth the trouble of its many dependencies to build it into PostGIS.
    cd /opt/installs
    tar xvf CGAL-4.4.tar.bz2
    cd CGAL_4.4
    cmake .
    

    (Note: in the 4.4 source, it was necessary to patch CGAL-4.4/src/CGAL_Qt4/all_files.cpp line to comment out line 2 with  // #include /opt/installs/CGAL-4.4/src/CGAL_Qt4/DemosMainWindow.cpp   to force not compiling the demos)

    sed -i '2 s/^/\/\//' src/CGAL_Qt4/all_files.cpp
    make
    make install
    
  20. Install gcc 4.7.2 from developer resource,  the SFCGAL team uses this newer-build C language compiler that provides syntax flexibility with ‘typename’ and they rely on it.  The leading part of environment variable PATH can be trimmed after compilation if desired to return to gcc 4.4.7 that comes with the CentOS 6.5 distribution
    cd /etc/yum.repos.d
    wget http://people.centos.org/tru/devtools-1.1/devtools-1.1.repo
    yum --enablerepo=testing-1.1-devtools-6 install devtoolset-1.1-gcc devtoolset-1.1-gcc-c++
    export CC=/
    export PATH=/opt/centos/devtoolset-1.1/root/usr/bin${PATH:+:${PATH}}
    
  21. Install SFCGAL from source,  the PostGIS wrapper for OGC objects in CGAL.  Simple Features in CGAL (SFCGAL) is an implementation of Open GIS Consortium (OGC) spatial object manipulations that can be accessed through an extended SQL syntax.
    cd /opt/installs
    tar xvf sfcgal-1.0.4.tar.gz
    cd SFCGAL-1.0.4
    cmake .
    make
    
  22. Install Apache Ant from binaries,  depends on Open JDK and java-devel installed at Step 2.  The Ant installer is a build tool created in Java, and it is used by PostGIS to build some drivers that are written in Java.
    Since it’s also needed for building Tomcat later, share it with a symbolic link in /bin.

    cd /opt/installs
    tar xvf apache-ant-1.9.4-bain.tar.bz2
    export ANT_HOME=/opt/installs/apache-ant-1.9.4
    PATH=$PATH:/opt/installs/apache-ant-1.9.4/bin
    export PATH
    ln -s /opt/installs/apache-ant-1.9.4/bin/ant /bin/ant
    
  23. Install pkg-config from source,  pkg-config is used at libffi build time to query the system’s installed libraries.
    cd /opt/installs
    tar xvf pkgconfig-0.18.tar.gz
    cd pkgconfig-0.18
    ./configure
    make
    make install
    
  24. Install libffi from source,  required for glib-2.0, this is the Foreign Function Interface library for C-language programming.  It allows code to dynamically call compiled functions by pointer rather than compiling the functions into the each module that uses it; elsewhere used in Python and Ruby.
    cd /opt/installs
    tar xvf libffi-3.1.tar.gz
    cd libffi-3.1
    ./configure
    make
    make install
    
  25. Install glib-2.0 from source,  required for GTK+ graphics, the GLib bundle is five C-language system libraries developed by GNOME project that provides generic memory management and threading and the GLib object system.
    cd /opt/installs
    xz -d glib-2.41.0.tar.xz
    tar xvf glib-2.41.0.tar
    cd glib-2.41.0
    ./autogen.sh
    ./configure --enable-man=no
    make
    make install
    
  26. Install GTK+  from source,  depends on Open JDK and java-devel installed earlier, and on the GLib object system.  The GNU Image Manipulation Program (GIMP) is a raster data editor.  The enhanced GIMP Tool Kit (GTK+) is a set of interface graphic widgets.
    cd /opt/installs
    xz -d gtk+-3.12.2.tar.xz
    tar xvf gtk+-3.12.2.tar
    cd glib-2
    
  27. Install Graph visualization toolkit, used by Doxygen for the PostGIS documentation generators.
    wget http://www.graphviz.org/pub/graphviz/stable\
    /redhat/el6/x86_64/os/graphviz-2.38.0-1.el6.x86_64.rpm
    
  28. Install Doxygen, used by the PostGIS documentation generators.
    cd /opt/installs
    tar xvf doxygen-1.8.7.src.tar.gz
    cd doxygen-1.8.7
    ./configure
    make
    make install
    ldconfig
    
  29. Install PostGIS from source,  finally this is the actual PostGIS build. Note that as of 2014.05.28 it was necessary to apply a patch according to this OSGeo Trac ticket so that PostGIS can build against the latest json-c 0.12, where the error calls being used were deprecated.su – postgres
    cd /opt/installs
    tar xvf postgis-2.1.3.tar.gz
    cd postgis-2.1.3
    ./autogen.sh
    ./configure –with-gnu-ld
    # hopefully yields something like the following:postgis_configureapply the patch per http://trac.osgeo.org/postgis/ticket/2723

    make -j2
    make check
    make install
    

    No fireworks to celebrate, just a shared object for PostgreSQL to use.
    postgis_listing

  30. Now it’s time to enable PostGIS in a first spatial test database.  This example calls it “sp_uno” and we’ll also verify that PostgreSQL is already running by trying to start it. ‘=#’  means the psql prompt.
    pg_ctl start
    createdb sp_uno
    createlang plpgsql sp_uno
    psql sp_uno
    
    =# CREATE EXTENSION postgis;
    =# CREATE EXTENSION postgis_topology;
    =# CREATE EXTENSION fuzzystrmatch;
    =# CREATE EXTENSION postgis_tiger_geocoder;
    =# \q
    

    For full ability to work with rasters, set these environment variables in the system’s global /etc/environment  file if they are always wanted.

    POSTGIS_ENABLE_OUTDB_RASTERS=1
    POSTGIS_GDAL_ENABLED_DRIVERS=ENABLE_ALL
    

    When creating a new ArcGIS enterprise geodatabase, these scripts should be run to enable PostGIS in the database (each separate database needs these initializations and they do not appear to conflict with ArcGIS use of the same database).
    Example: for a freshly created ArcGIS database egdb1, as system user postgres run

    cd /opt/installs/postgis-2.1.3
    psql -d egdb1 -f  doc/postgis_comments.sql
    psql -d egdb1 -f  spatial_ref_sys.sql
    psql -d egdb1 -f  postgis/sfcgal.sql
    

    that should leave 1068 functions in the PostgreSQL  public schema for db egdb1.

  31. Perform the PostGIS Garden Test to verify the install.  In the documentation directory /opt/installs/postgis-2.1.3/doc there are source files that can be used to generate PostGIS documentation, as well as some testing tools.  Based on instructions posted here, it’s possible to configure global tests.  If this is the first time running these tests, fear no error when dropping a nonexistent testpostgis db.  These commands will create a testing db, enable PostGIS within it, and run the two torture tests.  The geo_torturetest is rather punative:
    cd /opt/installs/postgis-2.1.3/doc
    xsltproc -o geo_torturetest.sql xsl/postgis_gardentest.sql.xsl postgis.xml
    xsltproc -o rast_torturetest.sql xsl/raster_gardentest.sql.xsl postgis.xml
    psql -U postgres -d postgres -c "DROP DATABASE testpostgis;"
    psql -U postgres -d postgres -c "CREATE DATABASE testpostgis;"
    psql -U postgres -d testpostgis -c "CREATE EXTENSION postgis;"
    psql -U postgres -d testpostgis -f ../topology/topology.sql
    psql -U postgres -d testpostgis -f ../postgis/sfcgal.sql
    psql -U postgres -d testpostgis -f geo_torturetest.sql > geo_torturetest_results.txt
    psql -U postgres -d testpostgis -f rast_torturetest.sql > rast_torturetest_results.txt
    

No responses yet

Jun 30 2014

SGeoS Add Esri Enterprise Geodatabase features – Module 4 of 9

Published by under SL In General

Add Esri Enterprise Geodatabase features

 

Build steps for configuration Module-stage-4

After PostgreSQL has been installed, the Esri ArcGIS for Server keycode produced when the ArcGIS for Server Standard application server was licensed for the machine can be used to enable an Esri Enterprise Geodatabase (the Application Formerly Known As ArcSDE) on just one of the PostgreSQL databases.

Natively, an Esri Enterprise Geodatabase will store geometry in its own implementation of ST_GEOMETRY type (distinct from Oracle’s implementation).  Esri ST_GEOMETRY is compact and optimized for fast exchange between ArcGIS and the geodatabase.  For spatial analysis with PostGIS, it is necessary to use the PostGIS implementation of OGC standards, PG_GEOMETRY type.  When there are performance concerns with ArcGIS Server, the ST_GEOMETRY type might be preferred as a publication format.  For spatial analysis with access to to CGAL geometry libraries, there are presently 1068 SQL functions in PostGIS, while there are 349 SQL functions in Esri SDE.  PostGIS can operate on features with PG_GEOMETRY, while Esri can operate on features with either PG_GEOMETRY or ST_GEOMETRY.  This creates a bit of complexity, yet also offers a number of ways to approach optimal storage.

When licensing Esri ArcGIS 10.2.2 for Server on the machine, the installation will create and store keycodes in a tiny WINdows Emulator provided by an open source application WINE.   The path to keycodes is like:

/ags1022/arcgis/server/framework/runtime/.wine/drive_c/Program Files (x86)/ESRI/License10.2/sysgen

1) Start from completed system Module-stage-3

2) Locate Esri Server License  As a licensed product, ArcGIS will require the server keycode to enable an Esri Enterprise Geodatabase.

To create a single new Esri Enterprise Geodatabase, use the Create Enterprise Geodabase tool
Esri_create_geodatabase

Choose Database Platform of PostgreSQL, set Instance to resolvable name for the server, define a new geodatbase name that is very short but descriptive, and supply your credentials for the postgres Database Admistrator, for the sde Geodatabase Administrator, and the path to keycodes for Authorization File.
Esri_Create_geodatabase2

To connect to an existing Esri geodatabase, use ArcCatalog or a Catalog pane in ArcMap, click Database Connections > Add Database Connection.
Esri_create_geodatabase3

In the Database Connection dialog, set Database Platform to PostgreSQL, for Instance use a resolvable name for the Geospatial Hybrid Device (GHD) server or its IP address, select Database authentication in the Authentication Type drop-down, and enter User name ‘sde’ and appropriate password, then select the Esri geodatabase from the Database drop-down.
Esri_create_geodatabase4

With a working connection such as “sde_to_eg1_on_sg11.sde”, open the connection by double-clicking its tin can icon tin_can_icon  then once open, right-click and follow context menu to Administration > Create and Manage Roles

No responses yet

Jun 30 2014

SGeoS Add Mono to provide MS .NET compatibility – Module 5 of 9

Published by under SL In General

Add Mono to provide MS .NET compatibility

Build steps for configuration Module-stage-5

1) Start from completed system Module-stage-4

2) Install from binary (ease of install is most important) Or go to next step
Mono is  a compromise, running a Microsoft .NET environment built for cross-platform use—but it provides a C# compiler and runs CIL in a way that will satisfy the needs of many .NET 4.5 applications.  For over 10 years, Mono developers worked to bring .NET to all manner of platforms, but in 2011 they formed a San Francisco-based startup Xamarin and have focused on producing commercial development tools that unify Windows, iOS and Android.  The platform has moved forward through version 3.2.8 of 2014.02.19 and yet with a mobile focus, the Xamarin startup may be headed toward acquisition, potentially even by Microsoft, weakening its long-term credibility for Open Source.  None the less, here’s how to add this component from binary to the Geospatial Hybrid Device.

Start with a tarball download

yum -y install glib2-devel libpng-devel libexif-devel libX11-devel fontconfig-devel
su – postgres
cd /opt/installs
wget http://download.mono-project.com/sources/mono/mono-3.2.8.tar.bz2
tar xvf mono-3.2.8.tar.bz2
cd mono-3.2.8
./configure --prefix=/usr/local

mono_autogen0

make
sudo make install
./configure --enable-nunit-tests
make -k check

Perhaps because the code base now includes iOS, Android, and Windows, it is expected that a few of the tests will not work.  A trivial C# program can be compiled and executed to validate:
cs_test_code

cs_test_code2

 
3) Clone Mono from git (less convenient; anticipates most current Mono version) These steps are repeated in Module 9 for the Open Simulator install, but placed here in case one prefers to emphasize most current Mono, at some extra effort.  Cloning a git repository is at the edge; if it doesn’t work one day, try again in a day or two when the developers have patched it up.

These steps are adapted from a blend of build descriptions
http://stackoverflow.com/questions/13184384/mono-3-0-0-build-on-centos-6
https://www.bluewhaleseo.com/blog/asp-netc-linux-centos6-apache2-ispconfig3-mono3/
http://stackoverflow.com/questions/22844569/build-error-mono-3-4-0-centos
http://stackoverflow.com/questions/11410020/compile-install-mono-on-centos-cant-get-past-make
while it’s compiling, mono looks like a mess of warnings for 20 minutes.

Building seems to work in the end, but it does evoke a certain Microsoft-like cloud of doubt while watching mono build.  When working, mono can be a very cool capability in the eyes of those invested in .NET apps and a lever for mitigating anti-Linux attitude among Windows developers.

cd /opt/installs
git clone  https://github.com/mono/mono.git
cd mono
./autogen.sh --prefix=/usr/local

mono_autogen0

The next step is the  intriguing “Use mono to make mono” step.  Monolite is the spartan build that only has enough to run the old gmcs.exe compiler, from which the real mono can be bootstrapped.

make get-monolite-latest
make EXTERNAL_MCS=${PWD}/mcs/class/lib/monolite/gmcs.exe -j 4
make check -j 4

Mono should pass all checks

make install
mono -V

mono_361_version

For future updates of Mono, if desired

cd /opt/installs/mono
git pull
./autogen.sh --prefix-/usr/local
make
make install

No responses yet

Jun 30 2014

SGeoS Add EAS Enterprise Address System Data Server Components – Module 6 of 9

Published by under SL In General

Add EAS Data Server components

Build steps for configuration Module-stage-6
1) Start from completed system Module-stage-5

2) Create partitions (or order VM properly with three virtual drives For production-grade installs, this is the motivation to have three separate virtual drives (one with three partitions on it, the other two single-partitioned).  The first drive has /boot, swap, and / root.  The second drive is used for PostgreSQL data. By having a distinct virtual drive with a single partition, if in the future more data space is required, then it will remain possible to extend the drive’s size. The third drive is for PostgreSQL logs, and in the same way is designed as a separate drive to always allow future enlargement.

Disk  sda  40GB
/dev/sda1        /boot    500 MB
(/dev/sda2)      swap    1x--2x dedicated system memory
/dev/sda3        /           ~35GB

Disk sdb  16GB
/dev/sdb1        /data    16GB for PostgreSQL data

Disk sdc  12GB
/dev/sdc1        /pg_xlog          12GB for PostgreSQL logs

No responses yet

Jun 30 2014

SGeoS Add EAS Enterprise Address System Geoserver components – Module 7 of 9

Published by under SL In General

While this module 7 of 9 may not have the same appeal as other similarly-named characters, assimilate the following steps to enable the Open Source Geospatial Foundation’s GeoServer as used by EAS.

Add EAS Geoserver components

Build steps for configuration Module-stage-7

Just a few more dependencies to deal with here.

1) Start from completed system Module-stage-6

2) Branded Java Install  The OSGeo GeoServer crew is partial to Oracle-branded Java, so download that from someplace near http://www.oracle.com/technetwork/java/javase/downloads/  then click through to Server JRE, and finally Linux x64.  Note that the last link will still require one to click to initiate the download—it doesn’t link directly to the desired file.  So, download to some Windows path since your server system won’t have a desktop.   Consider using the SMB share in /ags1022 a.k.a. O:\ on Windows to get the download easily in reach, and copy it over to /opt/installs along with the other downloads.

cp /ags1022/server-jre-8u5-linux-x64.gz /opt/installs/jdk-server-jre-8u5-linux-x64.gz
tar xvf jdk-server-jre-8u5-linux-x64.gz
cd jdk1.8.0_05
sudo alternatives --install /usr/bin/java java /opt/installs/jdk1.8.0_05/bin/java 3
sudo alternatives --config java

 

3) Microsoft TrueType core fonts install  Some fonts are desired, and for some reason EAS is partial to Microsoft fonts.  So install them already.

sudo yum localinstall msttcorefonts-2.5-1.noarch.rpm

4) Legacy Speed Tweak for Java-based Imaging   Next is Java Advanced Imaging, which helps speed up Geoserver.  This is a pretty old and stale-looking project, with no changes in 8 years, but the Geoserver wiki says they depend on it for better speed.  Its installation instructions may be hard to find.  If you’ve use the alternatives approach to access JDK 1.8.0 above, then the top part of the destination is shown below.  Otherwise, place the resources in the active/chosen JDK-directory/ jar/lib/ext

cd /opt/installs/jdk1.8.0_05/jre
wget http://download.java.net\
/media/jai/builds/release/1_1_3/jai-1_1_3-lib-linux-amd64.tar.gz
tar xvf jai-1_1_3-lib-linux-amd64.tar.gz
cd jai-1_1_3/lib
cp *.jar /opt/installs/jdk1.8.0_05/jre/lib/ext
cp *.so /opt/installs/jdk1.8.0_05/jre/lib/amd64

 

5) Prerequisites for a second Tomcat install  Once Java is emplaced, it’s time for another Tomcat; the first one was added as part of the ArcGIS 10.2.2 for Server silent install.  That first tomcat has already snagged a listen on :6080 and provides the pathway to ArcGIS Server resources.  This being a second one, and given the very recent version of Java just installed, let’s push forward with a late compatible version of Tomcat, where the installation version choices are discussed here.The ArcGIS for Server 10.2.2 has installed a reasonably current Tomcat, yet for the sake of isolation between Geoserver and Esri ArcGIS for Server and to pursue maximum Geoserver performance, we’ll try one major version step forward on both Java and its associated (second) Tomcat instance.

# cd /ags1022/arcgis/server/framework/runtime/tomcat/bin
# sh version.sh

Second_Tomcat

For this example, Tomcat 8.0.8 will be built in the /opt/installs location, after creating a user specific to the Tomcat major version being installed.

useradd tom8
passwd tom8

 

Now is the time to record this assignment in the run book

usermod -G tom8,installer tom8

 

If the entire /opt directory has group ownership of installer and 775 permissions, then tom8 should be able to create the /opt/tomcat/base directory and build Tomcat there.

# chgrp -R installer /opt
# chmod 775 /opt
ls -ld /op*

Tomcat_check

6) Build Tomcat 8 from source  Once ready, it’s time to set up for multiple Tomcats.   In this example, the Tomcat is built alongside other work in /opt/installs.  The deployments will be given named instances in numbered directories for ease of management.  As the first Tomcat (instance 0) listens to port :8080, the second (instance 1) will be configured to port :8180, (instance 2) to :8280, and so forth.  There are a few other control and redirect ports that will also scale by :+100.
But first, build a most-recent-stable Tomcat.  Get a source tarball (or is it a Tomcat hairball?) here.  The link to your chosen version number e.g. 8.0.8 is what to click through.  Build instructions based on this page, and multiple Tomcat instance deployments modeled from this page.

tomcat_download1
use path from the tar.gz link
tomcat_download2

su - tom8

cd /opt/installs
export ANT_HOME=/opt/installs/apache-ant-1.9.4
PATH=$PATH:/opt/installs/apache-ant-1.9.4/bin
export PATH

wget http://<some_apache.org_mirror>\
/apache/tomcat/tomcat-8/v8.0.8/src/apache-tomcat-8.0.8-src.tar.gz
tar xvf apache-tomcat-8.0.-src.tar.gz
cd apache-tomcat-8.0.8-src

Since this build is being done by the tom8 user, it is not permitted to build at the default system location /usr/share/java.  Avoid this faux pas by editing a proper, build.properties file.

cp  build.properties.default  build.properties

edit the value defined for base.path to something like /opt/tomcat/base
tomcat_path

ant

Ant will be downloading many jar files from Apache.org and  Eclipse.org over about a minute.
The dependencies will reside in /opt/tomcat/base, and the actual built Tomcat ends up down in /opt/installs/apache-tomcat-8.0.8-src/output/build   as a reference copy.
tomcat_check2
7) Boost performance with Apache Portable Runtime (APR) and Tomcat Native Library  As observed by the Tomcat configuration test, the best performance in production may be achieved by adding a native library.  This should provide some functions natively compiled faster than interpreted Java code, in particular SSL encryption.

wget http://<some apache mirror site>\
/apache//apr/apr-1.5.1.tar.gz
tar xvf apr-1.5.1.tar.gz
cd apr-1.5.1
./configure
make
sudo make install

This should place the APR configuration file at /usr/local/apr/bin/apr-1-config

wget http://<some apache mirror site>\
/apache/tomcat/tomcat-connectors/native/1.1.30/source/tomcat-native-1.1.30-src.tar.gz
tar xvf tomcat-native-1.1.30-src.tar.gz
cd /tomcat-native-1.1.30-src
cd jni
ant
cd native

8) Install two Tomcat instances in multi-instance configuration  With a clean deployment directory structure in mind, deployment is a matter of copying the reference directories from  build to a destination, and changing a few items in configuration files.  First copy into two instance directories; Instance 1 will be given to GeoServer and named geosrvr while Instance 2 will be kept available for the next Java servlet application and named tomtwo.

cp -pr  /opt/installs/apache-tomcat-8.0.8-src/output/build  /opt/tomcat/1
cp -pr  /opt/installs/apache-tomcat-8.0.8-src/output/build  /opt/tomcat/2

Then, make a (very) temporary install of a legacy Tomcat from distribution to grab its RHEL configs

yum -y install tomcat6
cp /etc/sysconfig/tomcat6  /opt/tomcat/base/tomcat6_sysconfig
cp /etc/init.d/tomcat6  /opt/tomcat/base/tomcat6_init_d
cp /etc/tomcat6/tomcat6.conf  /opt/tomcat/base/tomcat6_orig.conf
cp /usr/sbin/tomcat6  /opt/tomcat/base/tomcat6_sbin
yum remove tomcat6
rm -rf /usr/share/java/tomcat6*
rm -rf /usr/share/tomcat6
mkdir -p /var/cache/tomcat8/temp
chown -R tom8 /var/cache/tomcat8
chgrp -R tom8 /var/cache/tomcat8

Modify the  the  tomcat6_orig.conf  script with values like these

CATALINA_BASE="/opt/tomcat/1"
CATALINA_HOME="/opt/tomcat/1"
JASPER_HOME="/opt/tomcat/1"
CATALINA_TMPDIR="/var/cache/tomcat8/temp"
CATALINA_PID=”/var/run/tomcat8.pid”

Modify the  the  tomcat6_init_d  script with values like these

TOMCAT_CFG=”/etc/tomcat8/tomcat8.conf”
TOMCAT_SCRIPT=”/usr/sbin/tomcat8”

 

Then place these scripts where they need to go:

mkdir /etc/tomcat8
cp -p /opt/tomcat/base/tomcat6_orig.conf  /etc/tomcat8/tomcat8.conf
cp /opt/tomcat/base/tomcat6_sbin  /usr/sbin/tomcat8
cp /opt/tomcat/base/tomcat6_init_d  /etc/init.d/tomcat8

These config files are the key to making named Tomcat instances out of the numeric Tomcat instance directories.  By installing these copies, each can be modified to launch independent instances.  Here is an example for the first two.

ln -s /etc/init.d/tomcat8  /etc/init.d/geosrvr
ln -s /etc/init.d/tomcat8  /etc/init.d/tomtwo
cp  /opt/tomcat/base/tomcat6_sysconfig  /etc/sysconfig/geosrvr
cp  /opt/tomcat/base/tomcat6_sysconfig  /etc/sysconfig/tomtwo

Edit  /etc/sysconfig/geosrvr to point to the first Tomcat instance path by setting these lines. Uncomment them, or copy and uncomment the copied line as you choose.

JAVA_HOME=”/opt/installs/jdk1.8.0_05/bin/java”
JRE_HOME=”/opt/installs/jdk1.8.0_05/jre/bin/java”
CATALINA_BASE=”/opt/tomcat/1”
TOMCAT_USER=”tom8”
CATALINA_PID=”/var/run/geosrvr.pid”
CONNECTOR_PORT=”8180”
TOMCAT_LOG=”/opt/tomcat/1/logs/catalina.out”

Edit  /etc/sysconfig/tomtwo  to point to the first Tomcat instance path by setting these lines. Uncomment them, or copy and uncomment the copied line as you choose.

JAVA_HOME=”/opt/installs/jdk1.8.0_05/bin/java”
JRE_HOME=”/opt/installs/jdk1.8.0_05/jre/bin/java”
CATALINA_BASE=”/opt/tomcat/2”
TOMCAT_USER=”tom8”
CATALINA_PID=”/var/run/tomtwo.pid”
CONNECTOR_PORT=”8280”
TOMCAT_LOG=”/opt/tomcat/2/logs/catalina.out”

Secure the installation with a Java key store as along the lines of this discussion

$JAVA_HOME/bin/keytool -genkey -alias tomcat -keyalg RSA \
-keystore /opt/tomcat/1/conf/.keystore

cp  /opt/tomcat/1/conf/.keystore  /opt/tomcat/2/conf

Tune definitions in each instance’s  server.xml to be distinct.  There is virtue in continuing the pattern for each instance, should it ever be necessary to add yet other instances.  Recall that the Esri installation uses Tomcat ‘0’, and the new numbered instances configured here start with Tomcat ‘1’.  The pattern is to add (Instance*100) to the port numbers for each successive Tomcat instance.  The first two configurations can go like this:

Edit  /opt/tomcat/1/conf/server.xml  to have these (the Server tag is closed at end of file)

<Server port=”8105” shutdown=”SHUTDOWN”>
<Service name="Catalina1">

<Connector port=”8180” protocol=”HTTP/1.1”connectionTimeout=”20000”
redirectPort=”8543” />

<Engine name="Catalina1" defaultHost="localhost">
<Connector port=”8109” protocol=”AJP/1.3” redirectPort=”8543” />

Edit  /opt/tomcat/2/conf/server.xml  to have these (the Server tag is closed at end of file)

<Server port=”8205” shutdown=”SHUTDOWN”>
<Service name="Catalina2">

<Connector port=”8280” protocol=”HTTP/1.1”
connectionTimeout=”20000”
redirectPort=”8643” />

<Engine name="Catalina2" defaultHost="localhost">
<Connector port=”8209” protocol=”AJP/1.3” redirectPort=”8643” />

Test the installation with its bin/  configtest.sh  script to

su - tom8
cd /opt/tomcat/1/bin
./configtest.sh

Manage the installation as tom8 through their scripts if the standard “service” install is tedious.
NOTE:  using this method, be certain to start up this service as user tom8, not as root

/opt/tomcat/1/bin/startup.sh
/opt/tomcat/1/bin/shutdown.sh

/opt/tomcat/2/bin/startup.sh
/opt/tomcat/2/bin/shutdown.sh

Edit  /opt/tomcat/1/conf/tomcat-users.xml  and  /opt/tomcat/1/conf/tomcat-users.xmlto add an administrative user who can access the manager GUI interface for each of the Tomcat instances

<role rolename="manager-gui"/>
<role rolename="admin-gui"/>
<user username="tom_admin" password="secret" roles="manager-gui,admin-gui,tomcat"/>

Edit  /opt/tomcat/1/conf/server.xml  to comment out the direct http Connector and create an SSL connector to use as the new default

<!--
<Connector port="8180" protocol="HTTP/1.1"
connectionTimeout="20000"
redirectPort="8543" />
-->

<Connector protocol="org.apache.coyote.http11.Http11NioProtocol"
port="8543"
maxThreads="150" SSLEnabled="true" scheme="https" secure="true"
keystoreFile="/opt/tomcat/1/conf/.keystore" keystorePass="secreto"
clientAuth="false" sslProtocol="TLS" />

Edit  /opt/tomcat/2/conf/server.xml  to comment out the direct http Connector and create an SSL connector to use as the new default

<!--
<Connector port="8280" protocol="HTTP/1.1"
connectionTimeout="20000"
redirectPort="8643" />
-->

<Connector protocol="org.apache.coyote.http11.Http11NioProtocol"
port="8643"
maxThreads="150" SSLEnabled="true" scheme="https" secure="true"
keystoreFile="/opt/tomcat/2/conf/.keystore" keystorePass="secreto"
clientAuth="false" sslProtocol="TLS" />

 

9) Install Apache Maven from source [Only required if building Geoserver from source] Building GeoServer from source depends on Apache Maven, a Java project management and build tool.  It’s possible to build Maven like this; uses Ant, which crawls slowly for 9 minutes.  Don’t set the build directory to be within the extracted source.

cd /opt/installs
wget http://<<some apache.org mirror site>>\
/apache/maven/maven-3/3.2.1/source/apache-maven-3.2.1-src.tar.gz
tar xvf apache-maven-3.2.1-src.tar.gz
cp -rp apache-maven-3.2.1 apache-maven-3.2.1-src
rm -rf apache-maven-3.2.1
cd apache-maven-3.2.1-src
export M2_HOME=/opt/installs/apache-maven-3.2.1
ant

maven1o2

export M2=$M2_HOME/bin
export PATH=$M2:$PATH

mvn --version

maven2o2

10) Build Geoserver from Git Clone  (or…)  Source is only for a development build.
Greater deployment efficiency will be found with a stable deployment from WAR (below).

mkdir /opt/installs/geoserver
cd /opt/installs/geoserver
git clone https://github.com/geoserver/geoserver.git
cd /opt/installs/geoserver/src
mvn clean install

 

Maven does a remarkably good job of building GeoServer and then running a great many tests on the resulting application.  After all, this is the developer’s version so you’d hope it’s well tested.
If there’s no patience for fixing Java version dependencies in some of the tests, just use guidance from http://docs.geoserver.org/latest/en/developer/maven-guide/index.html

mvn -DskipTests clean install
mvn clean install -P restconfig

 

11) Install Geoserver from WAR  The source web app archive (WAR) is available in .zip format.

cd /opt/tomcat/1/webapps
wget http://sourceforge.net/projects/geoserver/files/GeoServer/2.5.1/geoserver-2.5.1-war.zip
unzip  geoserver-2.5.1-war.zip
cd /opt/tomcat/1/bin
./shutdown.sh
./startup.sh

 

12) Observe Geoserver in Tomcat Admin GUI  Tomcat has a built-in web administration app, but one must first configure a Tomcat admin user who can access it.  This can be done by editing the user file  /opt/tomcat/1/conf/tomcat-users.xml and adding sections like

<role rolename="manager-gui"/>
<user username="tom_admin" password="*******" roles="manager-gui,tomcat"/>

Then navigate a browser to the Tomcat instance, such as https://sg11:8543 and click the button in the upper right labeled Manager App
tomcat_admin1

Once logged in, the screen should show that Tomcat container instance’s running applications
tomcat_admin2
13) Localize the Geoserver Install  Clicking on the path link to /geoserver in the Tomcat Web Application Manager, one should get to the running Geoserver Welcome page.  If you are provoked by the defaults to localize, the Welcome page can be tuned by editing     /opt/tomcat/1/webapps/geoserver/data/global.xml

More importantly, prepare production use by moving data directory to a separate location per http://docs.geoserver.org/stable/en/user/production/data.html by first creating a new top-level data direcotry for Geoserver 2.5.1

cd /opt/tomcat/1/webapps/geoserver
sudo mkdir /gdata
sudo chown tom8 /gdata
sudo chgrp tom8 /gdata
cp -r data /gdata

then involves editing /opt/tomcat/1/webapps/geoserver/WEB-INF/web.xml
To include an uncommented section like this

<context-param>
<param-name>GEOSERVER_DATA_DIR</param-name>
<param-value>/gdata/data</param-value>
</context-param>

Save the edits, and then either Reload, or Stop then Start the geoserver application
tomcat_geoserver_app

It’s then possible to delete the old data directory  /opt/tomcat/1/webapps/geoserver/data

14) Verify Geoserver runs properly  Just browse to the geoserver app’s home page and explore some of Data > Layer Preview items

https://sg11:8543/geoserver/

No responses yet

Jun 27 2014

SGeoS Add EAS Enterprise Address System Web App Server components – Module 8 of 9

Published by under SL In General

Add EAS Web Application Server components

Build steps for configuration Module-stage-8

These final sections are adapted from deployment notes here
https://sfgovdt.jira.com/wiki/display/MAD/linux+deploy
https://sfgovdt.jira.com/wiki/display/MAD/web+server+-+setup+and+admin+-+linux
At this late point in the module assembly, there should already be a new Python 2.7.6 environment available

1) Start from completed system Module-stage-7

2) Verify access to updated Python  Since the Centos 6.5 system default python 2.6.6 remains intact, activate the updated by invoking the virtual environment created earlier

source /opt/installs/venv/bin/activate
python --version
   Python 2.7.6

3) Enable httpd server-info for EAS debugging   By default this will (rightly) be disabled. These directions are adapted from http://httpd.apache.org/docs/current/mod/mod_info.html
Edit /usr/local/httpd/conf/httpd.conf to uncomment these lines

LoadModule authz_core_module modules/mod_authz_core.so
LoadModule authz_host_module modules/mod_authz_host.so
#   disable ASAP after EAS debugging
LoadModule info_module modules/mod_info.so

and also add a section like this at the bottom of /usr/local/httpd/conf/httpd.conf

<Location /server-info>
   SetHandler server-info
   Allow from 10.x.xx.108
</Location>

then navigate a browser to the new location https://sg11/server-info

4) Build mod_wsgi from source   This is the mighty Web Server Gateway Interface module for Apache httpd that lets Python in; steps below adapted from the project documentation at https://code.google.com/p/modwsgi/wiki/QuickInstallationGuide
When a migration was forced from code.google.com, Graham Dumpleton appears to have created the new distribution point on GitHub.

cd /opt/installs
wget -O mod_wsgi-4.2.3.tar.gz \

https://github.com/GrahamDumpleton/mod_wsgi/archive/4.2.3.tar.gz

tar xvf mod_wsgi-4.2.3.tar.gz
cd mod_wsgi-4.2.3
./configure --with-apxs=/usr/local/httpd/bin/apxs --with-python=/usr/local/bin/python2.7
make

5) Install mod_wsgi as httpd module   Done as root.

cd /usr/local/httpd/modules
cp /opt/installs/mod_wsgi-4.2.3/src/server/.libs/mod_wsgi.so  .

Edit  /usr/local/httpd/conf/httpd.conf to include the following line in the general vicinity of line 153  below similar statements, and load  wsgi_mod

LoadModule wsgi_module modules/mod_wsgi.so

Restart the newly modded web server

apachectl2  restart

verify the richness of the new wsgi_mod by browsing to the location like
https://sg11/server-info#mod_wsgi.c

6) Build Psycopg from source and install   Done as system user  postgres after making the upgrade to Python 2.7.6 a default choice

cp  /usr/bin/python  /usr/bin/python2.6
rm  /usr/bin/python
ln -s /usr/local/bin/python2.7  /usr/bin/python
su - postgres
cd /opt/installs
wget  https://pypi.python.org/packages/source/p/psycopg2/psycopg2-2.5.3.tar.gz
cd psycopg2-2.5.3

In the psycopg directory, edit setup.cfg to point to PG configuration script pg_config
by uncommenting and setting this line to point at the postgresql built from source NOT using the –with-gssapi  configure option.

pg_config=/usr/local/pgsql/bin/pg_config
static_libpq=1

then build the module, and install

python2 setup.py build
sudo python setup.py install

then verify the psycopg2 version using the newer Python.
psycopg_version

 
7) Confirm PROJ.4 and GEOS installs    Already done as Module-Stage-3, Step 4 and Step 5 for PROJ.4 and GEOS, respectively.

8) Drop in Django    Done as root to load as a resource in system Python.  Uses pip installer that was installed during  Module-Stage-1, Step 9, which makes the process exceedingly easy.

pip install Django==1.6.5

Then run python, perhaps as system user postgres to confirm the install
django_version

9) Get Jogging  The python logging wrapper–pronounced “yogging”

cd /opt/installs
wget -O zain-jogging-0.2.2.tar.gz https://github.com/zain/jogging/tarball/v0.2.2
tar xvf zain-jogging-0.2.2.tar.gz
cd zain-jogging-976ff35
python2 setup.py install

10) Install EAS source  Clone the EAS source from Atlassian Bitbucket repository. The following example describes the URL for bitbucket user “RacerX”, then saves a copy of the clone directory with a date stamp.  The cloning time might be over 10 minutes, so

cd /opt/installs
hg clone https://RacerX@bitbucket.org/sfgovdt/eas
cp -r eas eaas_clone_yyyymmdd_hhmm
cd /opt/installs/eas

Initial deployment involves copying the eas folder to root of httpd web content.  Here the copy is named with trailing ‘0’ for testing purposes.

cd /opt/installs
cp -r eas /htdocs/eas0

11) Grab latest stable OpenLayers  The Open Source Geospatial Foundation’s OpenLayers Project provides a JavaScript API for open maps that is used by EAS.

cd /opt/installs
wget http://openlayers.org/download/OpenLayers-2.13.1.tar.gz
tar xvf OpenLayers-2.13.1.tar.gz

12) Obtain  ExtJS  The JavaScript object framework produced by Sencha Inc. for efficiently crafting very high-quality interfaces.  This is a commercial product that can be obtained by agreeing to terms of the Gnu Public License.

cd /opt/installs
wget http://cdn.sencha.com/ext/gpl/ext-4.2.1-gpl.zip
unzip ext-4.2.1-gpl.zip

13) Deploy EAS web application code  The deployment of EAS web app involves copying code to a chosen directory in Tomcat / webapps.  Part of the process has been automated, but it is not detailed here in the system build procedure.

No responses yet

Jun 26 2014

SGeoS OpenSim 0.8 on Mono 3.6.1 in CentOS 6.5 for SGeoS (Standard Geospatial Server) – Module 9 of 9

Published by under SL In General

As written yesterday, I’m working my way back through build notes.  This draft shows how I reached a couple of dead ends, backed up and took a small turn.  Here’s the end point where new Open Simulator 0.8, not quite two weeks old, is on a git clone of yesterday’s Mono.
The preceding steps have been written as they were developed–and will be posted subsequently.

Install OpenSim to Demonstrate Mono Apps

Build steps for configuration Module-stage-9

 

This installs the immersive 3D virtual world Open Simulator, developed as a C# application, to demonstrate the use of mono on the SGeoS testbed.  It also provides an update to the latest possible mono version, an ASP.NET module for Apache, and the curious little web server XSP as a more direct way of deploying .NET applications.

1)   Start from completed system Module-stage-8


2)  Verify prerequisites and build libGDI+  These are adapted from a build description page http://stackoverflow.com/questions/13184384/mono-3-0-0-build-on-centos-6

 yum -y  update
 yum -y install libpng-devel libjpeg-devel giflib-devel libtiff-devel libexif-devel libX11-devel \
     fontconfig-devel gettext httpd-devel

Confirm that the system standard pkg-config is used from /usr/bin/pkg-config or if any updates have been made, copy them elsewhere and link to /usr/bin/pkg-config; the make can take 12 minutes to build on a single thread, and using the -j4 thread saves nine minutes on an Atom host.

 which pkg-config
 /usr/local/bin/pkg-config
 cd /opt/installs
 wget http://download.mono-project.com/sources/libgdiplus/libgdiplus-2.10.9.tar.bz2
 tar xvf libgdiplus-2.10.9.tar.bz2
 cd libgdiplus-2.10.9
 ./configure --prefix=/usr/local
 export echo=echo
 make -j 4
 make install
 ldconfig

3)  Build mod_mono   This is an Apache connector for .NET requests.   It does not appear to be tuned for use with Apache 2.4, but can get along fine with CentOS 6.5 default Apache 2.2.16

Build instructions adapted from
https://www.bluewhaleseo.com/blog/asp-netc-linux-centos6-apache2-ispconfig3-mono3/
and requires axps which appears from httpd-devel group.

 cd /opt/installs
 wget http://download.mono-project.com/sources/mod_mono/mod_mono-2.10.tar.bz2
 tar xvf mod_mono-2.10.tar.bz2
 cd mod_mono-2.10
 ./configure prefix=/usr/local
 make
 make install
 ldconfig

To use the module, edit the relevant httpd.conf (likely either /etc/httpd/conf/httpd.conf or /usr/local/httpd/conf/httpd.conf)   to add this line to load mod_mono module and associate all the typical Microsoft web serving stuff, like index.aspx, Default.aspx, default.aspx, and associate ASP.NET file extensions with the ASP.NET MIME type.

 Include /etc/httpd/conf/mod_mono.conf

If, instead of all that, one simply want the mod_mono module alone to be loaded, then use this
LoadModule mono_module /usr/lib64/httpd/modules/mod_mono.so

4)  Clone Mono from git   There appeared to be issues with the very recent release tarballs, and for deployment of department apps, Mono could be an important part of the system.  These steps have been blended from several build descriptions
http://stackoverflow.com/questions/13184384/mono-3-0-0-build-on-centos-6
https://www.bluewhaleseo.com/blog/asp-netc-linux-centos6-apache2-ispconfig3-mono3/
http://stackoverflow.com/questions/22844569/build-error-mono-3-4-0-centos
http://stackoverflow.com/questions/11410020/compile-install-mono-on-centos-cant-get-past-make
while it’s compiling, mono looks like a mess of warnings for 20 minutes.

Cloning a git repository is at the edge; if it does not work one day, try pulling again in a day or two when the developers have patched it up.

While building works in the end, it does evoke a certain Microsoft-like cloud of doubt while watching mono build.  When it runs well, mono can be a very cool capability in the eyes of those invested in .NET apps and a lever for mitigating anti-Linux attitude among Windows developers.

 cd /opt/installs
 git clone  https://github.com/mono/mono.git
 cd mono
 ./autogen.sh --prefix=/usr/local

mono_autogen0

 

The next step is the  intriguing “Use mono to make mono” step.  Monolite is the spartan build that only has enough to run the old gmcs.exe compiler, from which the real mono can be bootstrapped.

 make get-monolite-latest
 make EXTERNAL_MCS=${PWD}/mcs/class/lib/monolite/gmcs.exe -j 4
 make check -j 4

Mono should pass all checks

 make install
 mono -V

mono_version_20140626

 

For future updates of Mono, pull from git if desired

 cd /opt/installs/mono
 git pull
 ./autogen.sh --prefix-/usr/local
 make
 make install

5)  Install Mono devleopment packages  these appear to be required by nant to build OpenSim.
These notes were made with reference to
http://stackoverflow.com/questions/16900575/install-mono-and-monodevelop-on-centos-5-x-6-x
It’s important that these Windows things know their way around so set the environmental variable such that PKG_CONFIG_PATH gets to the directory that’s got your latest mono.pc in it

 yum install gtk2-devel libglade2-devel
 export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig:\
 /usr/share/pkgconfig:/usr/lib/pkgconfig:/usr/lib64/pkgconfig

Go for gtk-sharp, a dependency of mono-develop

 cd /opt/installs
 wget http://download.mono-project.com/sources/gtk-sharp212/gtk-sharp-2.12.8.tar.bz2
 tar xvf gtk-sharp-2.12.8.tar.bz2
 cd gtk-sharp-2.12.8
 ./configure --prefix=/usr/local

Patch three sources in glib/glue/*.c  that try to include below <glib.h/> to only include <glib.h>
This was necessary on 2014.06.25 and may not remain so.
patch_1o2

change them to look like the following, then make; expect to see warnings and some sorts of errors.

patch_2o2

 make
 make install

Now go for gnome-sharp, another dependency of mono-develop

 yum install pango-devel atk-devel libgnome-devel libgnomecanvas-devel libgnomeui-devel
 cd /opt/installs
 git clone git://github.com/mono/gnome-sharp
 cd gnome-sharp
 ./bootstrap-2.24 --prefix=/usr/local
 make
 make install

Now go for mono-addins, yet another dependency of mono-develop

cd /opt/installs
 git clone git://github.com/mono/mono-addins
 cd mono-addins
 ./autogen.sh --prefix=/usr/local
 make
 make install

Finally go for mono-develop itself

cd /opt/installs
 wget http://download.mono-project.com/sources/monodevelop/monodevelop-3.1.1.tar.bz2
 tar xvf monodevelop-3.1.1.tar.bz2
 cd monodevelop-3.1.1
 ./configure --prefix=/usr/local
 make -j 4
 <!-- fail 2014.06.25>
cd /opt/installs
 git clone git://github.com/mono/monodevelop
 cd monodevelop
 ./configure --prefix=/usr/local --select

<< maybe just choose main >>

make
 <!-- fail 2014.06.25>

Don’t be too disheartened if the full monodevelop binary doesn’t build, as having the dependencies is a big part of the purpose here.

6)  Install Nant from binary  a popular MS adaptation of Apache Ant build tool.  Used for building Open Simulator or other complex .NET assemblies from source, if one wants to try that later.

cd /opt/installs
 wget http://sourceforge.net/projects/nant/files/nant/0.92/nant-0.92-bin.tar.gz
 tar xvf nant-0.92-bin.tar.gz

that extracts a working binary version of NAnt.exe, so make a script to invoke it

touch /usr/local/bin/nant

then edit that file to include the invocation against mono

mono /opt/installs/nant-0.92/bin/NAnt.exe

finally set the launch script to be executable and perhaps owned by installer group.
The test is ironic, yet informative.

chmod 754 /usr/local/bin/nant
 chown root:installer /usr/local/bin/nant
 nant

NAnt_version_20140626

 

7)  Install MySQL from repository  Just the basics; MySQL might even already be installed.  This is a preferred means of asset storage for Open Simulator, although not configured there by default.

yum install mysql

 

8)  Install Open Simulator binary  A build from source might be attempted, as CentOS doesn’t seem to have been the most popular choice among Opensim adherents.  Requires Mono >= 2.4.3 and NAnt >= 0.85 which both should be satisfied at this point.  A better choice: the compiled binary will very likely just run with Mono (!)

For a source build attempt, satisfy nant with a symlink at a deprecated location.

ln -s /usr/local/lib/mono/4.5/mcs.exe /usr/local/lib/mono/2.0/gmcs.exe

Then just get the OpenSim 0.8 binary; you’ve already got a very current Mono available

cd /opt/installs
 wget http://opensimulator.org/dist/opensim-0.8.tar.gz
 tar xvf opensim-0.8.tar.gz
 cd opensim-0.8
 cd bin

Back up default OpenSim.ini  to  OpenSim_orig.ini — and other files as appropriate from bin/config-include plus configurations described at http://opensimulator.org/wiki/Build_Instructions
To launch a configured OpenSim, it’s like this:

mono OpenSim.exe

Edit  /etc/sysconfig/iptables to have a line like this to allow OpenSim access on :9000

-A INPUT -m state --state NEW -m tcp -p tcp --dport 9000 -j ACCEPT

Then restart iptables

service iptables restart

Testing was conducted with the Singularity viewer for 64-bit Windows, with connections to the simulator running on SGeoS  http://www.singularityviewer.org/

SGeoS_OpenSim_first_light_20140626

In-world saved image of initial simulator region, demonstrating terrain edits, vegetation, object construction, and time-of-day adjustments.  Viewer and interactive editing from Windows 7 workstation, using Singularity Viewer (64-bit) 1.8.5 (5617) to  Open Simulator 0.8 server as compiled C# project running on SGeoS under Mono 3.6.1.

 

That’s it for this module’s draft.  The server, it’s working.  If one were to only care about getting OpenSim 0.8 running on Mono 3.6.1, then perhaps this is everything.  For the SGeoS, these build notes are only the ninth of nine modules.

No responses yet

Jun 25 2014

Open Simulator joins the SGeoS build — a strategy for blogging the builds

This is the first of what should be  a set of posts that detail a server build process for the San Francisco Enterprise Geographic Information Systems Program (SFGIS) Standard Geospatial Server (SGeoS).  In fact, the build work has been ongoing for several weeks and is concluding here, with OpenSim.

The motivation for including OpenSim in the platform was a desire to provide support for legacy .NET applications that may exist in various departments. In the interest of creating a Microsoft-neutral build that is framed with Open Source components, it was natural to bundle the Mono framework into the SGeoS design.  And while individual department applications are their own business and not part of the standard build, OpenSim serves as an excellent demonstration of the utility of the Mono framework as included on the server.  That , together with my perspective that immersive 3D clearly should be associated with geospatial servers, is why OpenSim is included in the Standard Geospatial Server.

OpenSim is not trivial by any means, and yet it is not such a resource hog that it would be infeasible to bundle it.  What’s more, it is an opportunity to distribute immersive 3D technology packaged with other geospatial capabilities.

Since the build descriptions are being transcribed from a build document that is approaching 80 pages on Google Docs, it seems prudent to break it up into individual modules.   And since WordPress here is configured to show older posts below newer ones—I’ll start down at the end modules and post new build descriptions for earlier modules in later days.

The original notion for SGeoS was to have modular build chapters that could provide a unit of capability.  That way, only selected modules need be configured.  After discussions with VMware engineers, I became intrigued by the notion of making a single server image that could run everything, all at once, and then disable unneeded featured in an actual deployment.  So the build document was initially structured with module-like chapters, but in fact the server builds them all—so it’s worth viewing the build document in sequence.

The modules will probably end up  numbering about 10, including packaging for production and possibly default-disabling of most items.   If one watches too closely, it might seem like I’m making a countdown to completion.  But this will end with a stub for deployment packaging, work back through an OpenSim build, and end up with imaging an install of CentOS 6.5 onto a new VM guest system.

No responses yet

Next »