Mind Chasers Inc.
Mind Chasers Inc.

PostgreSQL 13 from Git on Ubuntu Linux 18.04

Clone, build, and configure PostgreSQL Master Branch (13+) on Ubuntu Linux 18.04 and use C language or Python to interact with a basic database via the libpq library

Overview

PostgreSQL 13 has recently been released. If you're not familiar with PostgreSQL, it is a powerful, stable, and highly configurable open source relational database. Since we're developers & our database is a core, networked system function, we want to work with the latest release and its source to facilitate optimization, customization, and experimentation. Below is a basic guide on how we built and configured PostgreSQL 13 (master branch) from Git on Ubuntu Linux 18.04.

Install and Build from Git

Before we begin, let's see if we have PostgreSQL packages already installed on our Ubuntu 18.04 system. We show various ways below to detect whether remants of postgres exist on our system.

$ which psql
$ 

$ ps -e | grep post
$

$ dpkg -l | grep postgres
$ 

$ dpkg -l | grep libpq
$

$ ldconfig -p | grep libpq
$

If it is installed, you can perform a remove or purge. Note that a purge will remove the configuration files along with the packages.

$ sudo apt purge postgresql-client-common postgresql-common postgresql

Next let's clone the postgresql git repo. We'll follow the official postgreSQL documentation, and there's also a wiki page for Working with Git

$ mkdir -p /build; cd /build	

$ git clone git://git.postgresql.org/git/postgresql.git

$ cd postgresql/

We're going to work with the master branch, but we could have also chosen to checkout REL_13_STABLE:

$ git branch -r
  origin/HEAD -> origin/master
  ...
  origin/REL_13_STABLE
  ...
  origin/master


$ ls
aclocal.m4  config  configure  configure.ac  contrib  COPYRIGHT  doc  GNUmakefile.in  HISTORY  Makefile  README  README.git  src

We can see from the directory list above, that we're dealing with an autotools project (e.g., configure script).

Also, we always prefer to build outside the source directory whenever possible, so let's create a new build directory and run configure from there. Note that you may need to install some packages before configure will succeed (e.g., libreadline6-dev, systemtap-sdt-dev, zlib1g-dev, libssl-dev, libpam0g-dev, and python-dev). Also, make sure you have flex and bison installed. If you get stuck on finding a package missing during configure, then take a look at the Ubuntu Packages Search page

$ sudo apt install libreadline6-dev systemtap-sdt-dev zlib1g-dev libssl-dev libpam0g-dev python-dev bison flex

$ openssl version 	
OpenSSL 1.1.1  11 Sep 2018	# we're using Ubuntu's package version

$ mkdir build; cd build  	# /build/postgresql/build

$ ../configure --prefix=/opt/pgsql --with-python --with-openssl --with-pam --enable-debug --enable-depend

$ make
...
All of PostgreSQL successfully made. Ready to install.

Note that configure has many build and install options. Run ../configure --help for more information.

Next we need to install postgreSQL. Keep in mind that we chose to customize the install path using --prefix=/opt/pgsql/

$ make install  # no need for sudo since we took ownership of /opt
...
PostgreSQL installation complete.

Let's now do some checking and general housekeeping:

$ cd /opt/pgsql/
$ tree -d -L 2
.
├── bin
├── include
│   ├── informix
│   ├── internal
│   ├── libpq
│   └── server
├── lib
│   ├── pgxs
│   └── pkgconfig
└── share
    ├── extension
    ├── timezone
    ├── timezonesets
    └── tsearch_data

$ find . -name '*.conf.sample'
./share/pg_ident.conf.sample
./share/pg_hba.conf.sample
./share/postgresql.conf.sample
./share/pg_service.conf.sample

Now configure the dynamic linker to find our postgreSQL libraries:

$ cd /etc/ld.so.conf.d/

# edit libc.conf as root to append /opt/pgsql/lib
$ more /etc/ld.so.conf.d/libc.conf 
/usr/local/lib
/opt/pgsql/lib


$ sudo ldconfig

$ ldconfig -p | grep libpq
	libpqwalreceiver.so (libc6,x86-64) => /opt/pgsql/lib/libpqwalreceiver.so
	libpq.so.5 (libc6,x86-64) => /opt/pgsql/lib/libpq.so.5
	libpq.so (libc6,x86-64) => /opt/pgsql/lib/libpq.so

Basic Configure of PostgreSQL

We want to work with a separate user account, and we'll follow the convention of using the user account postgres.

$ sudo adduser --system --home /opt/pgsql/home  --shell /bin/bash --group postgres
Adding system user `postgres' (UID 123) ...
Adding new group `postgres' (GID 130) ...
Adding new user `postgres' (UID 123) with group `postgres' ...
Creating home directory `/opt/pgsql/data' ...

$ sudo adduser postgres ssl-cert
Adding user `postgres' to group `ssl-cert' ...
Adding user postgres to group ssl-cert
Done.

$ sudo adduser postgres sudo		# or optionally perform a visudo
Adding user `postgres' to group `sudo' ...
Adding user postgres to group sudo
Done.

$ sudo passwd postgres			# set our user passwd

$ su postgres
$ cd
$ pwd
/opt/pgsql/home

$ sudo chown -R postgres:postgres /opt/pgsql

$ /opt/pgsql/bin/initdb -D /opt/pgsql/data --pwprompt

The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.UTF-8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

Enter new superuser password: 
Enter it again: 

creating directory /opt/pgsql/data ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... America/New_York
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok

initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.

Success. You can now start the database server using:

    /opt/pgsql/bin/pg_ctl -D /opt/pgsql/data -l logfile start

Note the warning above about enabling trust authentication. Carefully review the documentation and setting in /opt/pgsql/data/pg_hba.conf

Starting the server and interacting with it

Before we start the server, let's modify the PATH environment variable to include /opt/pgsql/bin

$ cd /opt/pgsql/home

$ echo 'export PATH=$PATH:/opt/pgsql/bin' >> .bashrc	# use single quotes so not to expand $PATH

$ echo 'export PGDATA=/opt/pgsql/data' >> .bashrc

$ source ./.bashrc

$ pg_ctl start -l logfile
server starting

$ ps -e | grep postgres
20853 ?        00:00:00 postgres
20855 ?        00:00:00 postgres
...
 
$ psql
psql (14devel)
Type "help" for help.

postgres=# 

Some basic interaction on the command line:

# select version();
                                                version                                                 
--------------------------------------------------------------------------------------------------------
 PostgreSQL 14devel on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit

# \list
                                  List of databases
   Name    |  Owner   | Encoding |   Collate   |    Ctype    |   Access privileges   
-----------+----------+----------+-------------+-------------+-----------------------
 postgres  | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | 
 template0 | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres          +
           |          |          |             |             | postgres=CTc/postgres
 template1 | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres          +
           |          |          |             |             | postgres=CTc/postgres
           
     
(3 rows)

# \d
Did not find any relations.

# select 2+2;
----------
        4
(1 row)

#\q

Create an example database

Below we create a minimal database, table, and table entry.

$ psql 

# create role dbowner with createdb login;
CREATE ROLE

# create database testdb owner dbowner;
CREATE DATABASE

# \l
                                    List of databases
    Name     |    Owner    | Encoding |   Collate   |    Ctype    |   Access privileges   
-------------+-------------+----------+-------------+-------------+-----------------------
 testdb | dbowner | UTF8     | en_US.UTF-8 | en_US.UTF-8 | 
...

#\q

$ psql testdb -U dbowner

# CREATE TABLE users (
	uid integer PRIMARY KEY,
	name varchar(32)
);
CREATE TABLE

# INSERT INTO users VALUES (1, 'Tony');

# select * from users;
 uid | name 
-----+------
   1 | Tony

Access our database using the C language

For this next step, we'll rely on some guidance provided by the following documentation on postgresql.org:

Let's first build and run the testlibpq.c example provided in the above reference.

$ gcc  -O0 -g -I/opt/pgsql/include testlibpq.c -L/opt/pgsql/lib -lpq -o testlibpq

$ ./testlibpq

oid            datname        datdba         encoding       datcollate     datctype       datistemplate  datallowconn   datconnlimit   datlastsysoid  datfrozenxid   datminmxid     dattablespace  datacl         

12695          postgres       10             6              en_US.UTF-8    en_US.UTF-8    f              t              -1             12694          480            1              1663                          
16385          testdb         16384          6              en_US.UTF-8    en_US.UTF-8    f              t              -1             12694          480            1              1663                          
1              template1      10             6              en_US.UTF-8    en_US.UTF-8    t              t              -1             12694          480            1              1663           {=c/postgres,postgres=CTc/postgres}
12694          template0      10             6              en_US.UTF-8    en_US.UTF-8    t              f              -1             12694          480            1              1663           {=c/postgres,postgres=CTc/postgres}

Next let's create a simple C program that accesses our database and prints out the first row from our users table:

#include <stdio.h>
#include <stdlib.h>
#include <libpq-fe.h>


int main() {

    PGconn *conn = PQconnectdb("dbname=testdb");
    if (PQstatus(conn) == CONNECTION_BAD) {
        fprintf(stderr, "Connection to database failed: %s\n", PQerrorMessage(conn));
        PQfinish(conn);
        exit(1);
    }

    PGresult *res = PQexec(conn, "SELECT * FROM users");
    printf("id: %s\n", PQgetvalue(res, 0, 0));
    printf("name: %s\n", PQgetvalue(res, 0, 1));

    PQclear(res);
    PQfinish(conn);
    return 0;
}

Build it and run it (as postgres):

$ gcc -O0 -g  -I/opt/pgsql/include testlibpq2.c -L/opt/pgsql/lib -lpq -o testlibpq2

$ $ ./testlibpq2 
id: 1
name: Tony

Access our database using Python

For Python access, we'll turn to the popular, third-party, open source library Psycopg. As stated on its web site: "Psycopg is the most popular PostgreSQL adapter for the Python programming language. At its core it fully implements the Python DB API 2.0 specifications. Several extensions allow access to many of the features offered by PostgreSQL."

Psycopg2 wraps around the libpq library, which we used in the previous C programming examples. Below we clone psycopg2 and take a brief look at the source:

Note that we're performing the following steps as a typical Ubuntu Linux user and not postgres or root.

$ cd /build
$ git clone https://github.com/psycopg/psycopg2.git
$ cd psycopg2/

$ ls
AUTHORS  INSTALL  LICENSE   MANIFEST.in  psycopg         README.rst  setup.cfg  tests
doc      lib      Makefile  NEWS         psycopg2.cproj  scripts     setup.py   tox.ini

$ find psycopg/ -name *.c | xargs grep PQ
psycopg/cursor_type.c:            status = PQtransactionStatus(self->conn->pgconn);
psycopg/cursor_type.c:            status = PQTRANS_UNKNOWN;
psycopg/cursor_type.c:        if (status == PQTRANS_UNKNOWN || status == PQTRANS_INERROR) {
psycopg/cursor_type.c:        if (PQgetisnull(self->pgres, row, i)) {
...

Below we choose to install psycopg2 into /usr/local/lib/python3.6/dist-packages/

$ more /opt/pgsql/env-pgsql 
export PATH=$PATH:/opt/pgsql/bin

$ source /opt/pgsql/env-pgsql 

$ which pg_config
/opt/pgsql/bin/pg_config

$ python3 setup.py build

$ ls dist
psycopg2-2.8.5.dev0-py3.6-linux-x86_64.egg

$ sudo cp dist/psycopg2-2.8.5.dev0-py3.6-linux-x86_64.egg 

$ ls /opt/lib/python3.6/site-packages/psycopg2-2.8.5.dev0-py3.6-linux-x86_64.egg /usr/local/lib/python3.6/dist-packages/
$ python3 
> import psycopg2
> conn = psycopg2.connect("dbname='testdb' user='dbowner'")
> cur = conn.cursor()
> cur.execute("SELECT * FROM users")
> rows=cur.fetchall()
> rows[0]
(1, 'Tony')
> quit()

Additional References

Didn't find an answer to your question? Post your issue below or in our new FORUM, and we'll try our best to help you find a solution.

And please note that we update our site daily with new content related to our open source approach to network security and system design. If you would like to be notified about these changes, then please follow us on Twitter and join our mailing list.

Related articles on this site:

share
subscribe to mailing list:

Please help us improve this article by adding your comment or question:

your email address will be kept private
authenticate with a 3rd party for enhanced features, such as image upload
previous month
next month
Su
Mo
Tu
Wd
Th
Fr
Sa
loading