Month: June 2017

On Demand or On fly tiles Render

Hey All. Today I want to discuss the concept of tiles rendering.

There are two types of rendering the tiles.

  1. Rendering on demand- The tiles are render when you zoom or go to that location.
  2. Rendering on the fly- Pre-render the tiles without zooming it again and again.

The command to render the list of map tiles by sending the request to rendering daemon. It will pre-render the tiles.

$ render_list -m default -a -z 0 -Z 10

Output

debug: init_storage_backend: initialising file storage backend at: /var/lib/mod_tile
Rendering client
Starting 1 rendering threads
Rendering all tiles from zoom 0 to zoom 10
Rendering all tiles for zoom 0 from (0, 0) to (0, 0)
Rendering all tiles for zoom 1 from (0, 0) to (1, 1)
Rendering all tiles for zoom 2 from (0, 0) to (3, 3)
Rendering all tiles for zoom 3 from (0, 0) to (7, 7)
Rendering all tiles for zoom 4 from (0, 0) to (15, 15)
Rendering all tiles for zoom 5 from (0, 0) to (31, 31)
Rendering all tiles for zoom 6 from (0, 0) to (63, 63)
Rendering all tiles for zoom 7 from (0, 0) to (127, 127)
Rendering all tiles for zoom 8 from (0, 0) to (255, 255)

Here, the flag -m means long prompt. This command takes some time. While the command was executing, I got to know the maths behind the tiles at a particular level.

Assume zoom level is x then the tiles are render from (0,0) to
((2^x-1),(2^x-1))

Screenshot from 2017-06-19 20-36-47.png

This picture clearly shows what above explained.

Osm2pgsql Error

Today I was trying to import data to the database using latest version of openstreetmap-carto. But I got an error.

$ osm2pgsql –slim -d gis -C 2000 –hstore –number-processes=4 –hstore -S ~/osm/openstreetmap-carto/openstreetmap-carto.style ~/Downloads/india-latest.osm.pbf

Error

osm2pgsql: error while loading shared libraries: libproj.so.0: cannot open shared object file: No such file or directory

Then I locate this shared library.

$locate libproj.so

Output

/usr/lib/x86_64-linux-gnu/libproj.so
/usr/lib/x86_64-linux-gnu/libproj.so.9
/usr/lib/x86_64-linux-gnu/libproj.so.9.1.0

Then I created the symbolic link.

$ sudo ln -s /usr/lib/x86_64-linux-gnu/libproj.so /usr/lib/libproj.so.0

And the problem was solved.

Postgresql Commands

Today, I have downloaded map of a country and imported it into the postgresql database through osm2pqsql.

All the relations in the database were having millions of rows. So, while selecting all of them at once I was getting error.

First I here tell how to open the psql shell. If you are not sudo user. Don’t Worry. We have a solution for that also.

To open shell as a superuser.

$sudo -u postgres psql

It will prompt to psql command line. To connect to particular database

\c gis

That’s all.

If you are a normal user. Request to sudo user to create new role for you and create new database for you.

postgres=# create role tamil login password ‘tamil’;

postgres=# CREATE DATABASE mydatabase WITH OWNER = tamil;

Then you access that particular database with sudo power.

$ psql -h localhost -d mydatabase -U tamil

Then , Dr. H S Rai suggested me to limit the number of rows. From there I got a keyword Limit and searched relevant query. I was not having an idea about this query initially. Without wasting more time in telling my story I here directly code the query.

gis=# select * from planet_osm_nodes LIMIT 3;

Output

-[ RECORD 1 ]—
id | 15382126
lat | 115301682
lon | 849624182
tags |
-[ RECORD 2 ]—
id | 15382127
lat | 115310352
lon | 849612464
tags |
-[ RECORD 3 ]—
id | 15382129
lat | 115327713
lon | 849590461
tags |

Another postgresql query to clear the screen of the psql command line.

gis=# \! clear

You can also use shortcut  Ctrl +l

Command to check the extensions created in the postgres.

postgres=# \dx

Output

List of installed extensions
Name | Version | Schema | Description
———+———+————+———————————————————————
hstore | 1.3 | public | data type for storing sets of (key, value) pairs
plpgsql | 1.0 | pg_catalog | PL/pgSQL procedural language
postgis | 2.2.1 | public | PostGIS geometry, geography, and raster spatial types and functions
(3 rows)

I want to know what basically happen when we create an extension. By creating an extension “postgis” it creates the relation “spatial_ref_sys”. You can check it also.

new=# create extension postgis;

It created a table

new=# \dt

List of relations
Schema | Name | Type | Owner
——–+—————–+——-+———-
public | spatial_ref_sys | table | postgres
(1 row)

To retrieve the value of a first ten rows of column tag with particular key

gis=# SELECT tags->’name:hi’ from planet_osm_polygon limit 10;

COALESCE function will output the first not null value among the arguments.

gis=# SELECT coalesce(tags->’name:hi’,name) from planet_osm_polygon limit 10;

To play more with the column having datatype hstore you can follow the tutorial.

http://www.postgresqltutorial.com/postgresql-hstore/

Command to check database size.

\l+

 

 

Convert Shapefile to database, Vice Versa

Shapefile to Database

Today, I tried to convert shapefile to sql file. Ultimately to import it into the postgis database.

For it you should have shapefile downloaded. If not having you can download it from below.

$wget http://wsgw.mass.gov/data/gispub/shape/state/boundaries.zip
$unzip boundaries.zip

Tool used to convert shapefile to sql file is shp2pgsql.

$shp2pgsql -G BOUNDARY_POLY.shp > BOUNDARY_POLY.sql

Make sure you should be in the directory where shapefile located. Here, sql file has insert commands to the relation country. So there must be table created if your sql file don’t have created table.

new=# CREATE TABLE country(id integer NOT NULL, name text, boundary geography(MultiPolygon,4326),name_alias text, CONSTRAINT “countryId” PRIMARY KEY (id ));

Command to dump sql file into the database.

postgres@amisha:~/9.5/main$ psql new < BOUNDARY_POLY.sql

Database to shapefile

To convert database back to shapefile, I have used pgsql2shp.

$ pgsql2shp -f am3 -h localhost -u postgres gis “SELECT * from planet_osm_line where admin_level=’4′ and name=’Punjab'”

Output

Initializing…
Done (postgis major version: 2).
Output shape: PolyLine
Dumping: X [19 rows].

Thank you.

QGIS and JOSM Installation

Steps to install QGIS

  1. Go to apt sources file.

$sudo vim /etc/apt/sources.list

  1. Add these two lines at the end.

deb http://qgis.org/debian xenial main
deb-src http://qgis.org/debian xenial main

  1. Then download the package information from the sources by update command.

sudo apt-get update
sudo apt-get install qgis python-qgis qgis-plugin-grass

It’s installed. Now, you can browse to QGIS from dashboard. But I run it from terminal. It’s easy.

You can learn QGIS from following tutorials.

https://docs.qgis.org/2.8/en/docs/

JOSM Installation

It’s a one-line command only.

sudo apt install josm

You can learn JOSM from following tutorials.

https://www.google.co.in/search?client=ubuntu&channel=fs&q=josm+tutorial&ie=utf-8&oe=utf-8&gfe_rd=cr&ei=j01FWavgKtbmugTdvYHQAw

 

Upgrade Carto Compiler

Today, I tried to upgrade Carto Compiler.

To be acquainted with Cartocss, one can follow this tutorial.

http://tilemill-project.github.io/tilemill/docs/manual/carto/

The current version of carto compiler is

$ carto –version
carto 0.9.5 (Carto map stylesheet compiler)

To upgrade it through npm. You should have nodejs, npm installed in order to upgrade carto.

$ sudo npm install -g carto

After upgrading it, I again checked the version. But I got the error.

$ carto –version
/usr/bin/env: ‘node’: No such file or directory

To solve this error I created the symbolic link of node from nodejs.

$ sudo ln -s /usr/bin/

nodejs /usr/bin/node

Problem Solved. It is upgraded.

$ carto –version
carto 0.18.1 (Carto map stylesheet compiler)

 

Sixth Day- Information Security

CDAC

Its name of Institute, Center for Development of Advanced Computing(CDAC). CDAC offers different PG Diploma courses in field of Computer Science, Electronics, Security, Mobile Computing… Diploma in Advanced Computing (DAC) is a well known course offered by CDAC.

In order to make our IT infrastructure resilient(recoverable from adverse affect) against these threats, there is a need for cutting-edge Research and Development efforts in Cyber Security. C-DAC has been actively pursuing R & D in a number of sub-areas in Cyber Security domain.

Information Security Assurance

  1. Confidentiality of data- It means data is not read and manipulated by intruders.
  2. Data Integrity – The data send by sender is the data received by receiver.
  3. Authentication- The sender and receiver can’t deny of sending and receiving respectively. One of technique used is Digital Signatures.
  4. Non-Repudiation- It’s the term used in legal contracts where sender or receiver later on can’t refuse to the authentication of the signatures.
  5. Non-Fabrication- The unauthorized person is sending the data to the receiver. It seems to receiver that data is sent by authorized sender which is untrue.
  6. Availability- The data is readily available when it is required.

Information security process is a three wheeler process.

CIA- Confidentiality  Integration Availability

Attack on 7 layer OSI Model

678d21456a59ea41bbe2f227429bc5b1

Common Online Attack

Denial of Service

It is cyber attack where the intruder make the machine resources unavailable to the intended users by temporarily or indefinitely disrupting(break) the services of host connected to internet.

DDoS Implementation

It is implemented through Botnet. Bot means an automated computer program to perform a particular task.

Here, Botmaster(intruder) cunningly insist people to install bot in their computer over a network. It can be through various ways like “Click here to win iphone7”;). When they click bot in their local machine is installed. Likewise over a network it is installed in millions of machines. Then botmaster orders the bot to go to particular site at a time(like Amazon). There is increase in the overhead on the amazon for that time. Hence intended users not able to use Amazon i.e. Denial of shopping:D

That is why it is Distributed denial of Service as it is done with the help of millions o computers cunningly.

Social Engineering

Social engineering principally involves manipulation of the people rather than technology to breech the security. Here human as a weapon for intruder to attack human.

Types of Social Engineering

Dumpster Diving- The intruder search the credentials and gather information from the dump or dustbin. Here, dustbin means the data which we throw in the trash.

Persuasion- It’s a telephone calling. Intruders persuade humans by calling them supposedly from banks. They play tactics to intake the credential information.

Shoulder Surfing- In this attack the intruder see confidential data by looking over the victim’s shoulder. Either by standing close to him or using binoculars.

Baiting- Baiting is when an attacker intentionally leaves a malware-infected physical device, such as a USB flash drive in a place it is sure to be found. The finder then picks up the device and loads it onto his or her computer, unintentionally installing the malware.

Phishing- Phishing is when a malicious party sends a fraudulent email disguised as a legitimate email, often purporting to be from a trusted source. The message is meant to trick the recipient into sharing personal or financial information or clicking on a link that installs malware.

Identity Theft- In identity theft the attacker use someone’s identity to do a crime or wrongful thing. It can be implemented through sim cards, BPO(Business Process Outsourcing).

A logic bomb is a piece of code intentionally inserted into a software system that will set off a malicious function when specified conditions are met. For example, a programmer may hide a piece of code that starts deleting files at a pre-defined time or when some other condition is met.

Some viruses attack their host systems on specific dates, such as Friday the 13th or April Fools’ Day. Trojans that activate on certain dates are often called “time bombs”.

That is why we say that update your Antivirus Software regularly.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Saturday Meet

Hello Everyone. Today was the first Saturday meet of 6-months training. It was very different from other Saturday’s as we have not given presentation on our daily progress rather delivered interactive lecture from one of the GD-member <Harmanpreet Singh>.

He told us about how to efficient your work even in the entirely different environment. He does job in Chennai in XYZ start-up company. He is the only one from Punjab and sometime face problem while communication. That company has a strict schedule and norms which are to be followed by everyone. Moreover, all over there are fan of Windows;)

<quote>Working in small start-up company is directly proportional to the learning or exploring yourself</quote>

Today, I came to know how GD started. It was because of three GD Members Mr. Vikas Mahajan and two more. They completed their training from Dr. Rai Sir only.

Coming to the technical work. Today, I solved the issue of brightness in Ubuntu 16.04 LTS.

After upgrading Ubuntu to 16.04 I was not able to change the brightness of my laptop.

Solution

sudo add-apt-repository ppa:apandada1/brightness-controller
sudo apt-get update
sudo apt-get install brightness-controller

It’s a work of three commands and you are done.

 

Installing Postgis in Ubuntu 16.04 LTS

Gear up Guys. Here, I am going to discuss new problem faced few days back.

While selecting data from table in gis database I was getting an error.

could not access file "$libdir/postgis-2.1": No such file or directory

Then I tried to install postgis-2.1 but I got a dependency error.

$ sudo apt-get install postgresql-9.3-postgis-2.1

The following packages have unmet dependencies: postgresql-9.3-postgis-2.1 : Depends: libgdal1h (>= 1.9.0) but it is not installable Depends: libgeos-c1 (>= 3.4.2) but it is not going to be installed Depends: liblwgeom-2.1.7 (>= 2.1.6) but it is not going to be installed Depends: libproj0 (>= 4.8.0-1) but it is not installable E: Unable to correct problems, you have held broken packages.

But when I tried to install dependencies, I got an error.

libgeos-c1v5 : Breaks: libgeos-c1 (< 3.4.2-8~) but 3.4.2-4ubuntu1 is to
be installed.

After browsing a lot, I was not able to resolve the problem. So I decided to upgrade Ubuntu from 14.04 to 16.04.

The command to upgrade Ubuntu is

$sudo do-release-upgrade

Then again I tried to install postgis.

$sudo apt-get update

After upgrading I got a notice. I was not able to install any package through apt-get.

N: Ignoring file '50unattended-upgrades.ucf-dist' in directory '/etc/apt/apt.conf.d/' as it has an invalid filename extension

To resolve it first check the difference between old file and upgraded file. If no difference then remove old file from there.

diff /etc/apt/apt.conf.d/50unattended-upgrades.ucf-old /etc/apt/apt.conf.d/50unattended-upgrades
sudo rm /etc/apt/apt.conf.d//50unattended-upgrades.ucf-old

Then again resynchronize the package index file by running update command.

$sudo apt install postgresql postgresql-contrib postgis postgresql-9.5-postgis-2.2

It will output like this in between.

Ver Cluster Port Status Owner Data directory Log file
9.5 main 5434 down postgres /var/lib/postgresql/9.5/main /var/log/postgresql/postgresql-9.5-main.log
Setting up postgresql (9.5+173) …

Congratulations, You have installed it unless in between no problem arrived. Here I am going to make osm server in Ubuntu 16.04.

I am following this link. I will describe that steps where you might face an error. Rest all is same.

All steps are almost same as discussed in previous blogs.

While logging into the psql command line I was getting an error.

amisha@amisha$sudo -u postgres psql
Error
psql: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket “/var/run/postgresql/.s.PGSQL.5432”?

After searching I came to know that I have to restart the postgresql.

$sudo service postgresql restart

After that, I tried to create extension postgis which gave an error.

postgres=# create extension postgis;

ERROR: could not open extension control file “/usr/share/postgresql/9.6/extension/postgis.control”: No such file or directory

This error came because I have postgresql-9.6 as well as postgresql-9.5. By default data_directory was 9.6 and there was not postgis.control file in it. Moreover it was present in the directory 9.5. Then I decided to change the default data directory.

I used this link to change it. They have explained it very nicely.

I want to change the data directory from /var/lib/postgresql/9.6/main to /var/lib/postgresql/9.5/main

All the commands written below are explained it in the link.

sudo systemctl stop postgresql
sudo systemctl status postgresql
sudo mv /var/lib/postgresql/9.6/main /var/lib/postgresql/9.6/main.bak

Now open the postgresql configuration file in your favourite editor. Mine is vim.

sudo vim /etc/postgresql/9.6/main/postgresql.conf

Look at the value of data directory here. Change it new value where you want like I want in the directory 9.5.

postres=# SHOW data_directory;

. . .
data_directory = ‘/var/lib/postgresql/9.6/main
. . .

Now start the postgresql again. You may check the status.

$sudo systemctl start postgresql

$sudo systemctl status postgresql

Now try to run the server again. Here, I got an error. Notice carefully there is difference between the two errors on a same command. Earlier there was no file or directory but now permission is denied.

$sudo -u postgres psql
psql: could not connect to server: Permission denied
Is the server running locally and accepting
connections on Unix domain socket “/var/run/postgresql/.s.PGSQL.5432”?

Then I saw the port number to which postgresql currently is listening through the command given below.

$ netstat -tulpn | grep postgres

Output

(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp 0 0 127.0.0.1:5434 0.0.0.0:* LISTEN 14571/postgres

There are two ways t solve this problem.

  1. To use port number every time.
  2. Change your port number permanently.

Using the first approach I ran the same command with port number popped up in the command.

$ sudo -u postgres psql -p 5434

I am able to login. Now able to create postgis extension also.

postgres=# create extension postgis;
CREATE EXTENSION

But this approach can bring problem again as we have done change on the terminal not permanently.

So, to permanently change port.

  1. Stop postgresql server.
  2. Make change in the configuration file like here. Change the port number for which the postgresql server searching for. Save it.

$sudo vim /etc/postgresql/9.5/main/postgresql.conf

$ sudo systemctl enable postgresql

Then again start the postgresql server.

The furthur explanation I will explain in my next blog.

Thank You

Fourth Day- Latex

Typesetting- It’s the arrangement and setting of the type. It’s a designing of formatted text. It’s a concept.

Typography- It’s a technique of arrangement of the type. Basically, it is used to implement typesetting.

File management or documentation is significance part of any project.

There are two types of typesetting software.

  1. Open source software- Examples-> Latex, Scribus, laidout
  2. Commercial Software- Examples-> MS Word, Adobe Indesign, QuarkXPress, istudio publisher.

Why programming is better than GUI?

  1. Automation.
  2. Take less time.
  3. Manipulate the things easily.
  4. Programming give real power whereas GUI has limited scope.

MS Word provides templates to make a good product.

Advantages of Latex

  1. Space optimization.
  2. It do less hyphenation(split words with – character)
  3. TeX combines a number of single/separate characters into a single ligature. Example with fi.

TexLive provides full packages comparable to Miktex.

TexStudio is just IDE. For backend you need Latex distribution like texlive or miktex.

Encoding

Encoding is the process of converting data into a format required for a number of information processing needs, including:

  • Program compiling and execution
  • Data transmission, storage and compression/decompression
  • Application data processing, such as file conversion
  • In computer technology, encoding is the process of applying a specific code, such as letters, symbols and numbers, to data for conversion into an equivalent cipher.

Vector Images and Raster Images

Vector images are the digital images created through a sequence of commands or mathematical statements that place lines and shapes in a given two-dimensional or three-dimensional space. Image extension like pdf, SVG,  eps etc.

Raster images are the digital images created or capture(scanning) as a set of samples of a given space. Raster image files are usually larger than vector image files. Image extension like png, jpg, jpeg etc.

MS Visio and Dia(open-source) are used to design images.

Since all basic are clear, so come to Latex now.

I will discuss here few basic commands.

In Latex, figures and tables are floats. Latex placed itself according to the optimized algorithm. But still if you want to the that position then add [H] with \beginfigure

\beginfigure[H]  where H means to keep figure here

Here, braces are optional. In place of H we can write T(for top), B(Bottom), P(Page)

\usepackage{subcaption}  //To insert 4 -5 images in the in one fig like a b c

Figure~  //Tilt sign will deoptimize space like I want figure-3.6 together then use tilt sign

\usepackage{hyperref} //it will take the pointer of the section in an index to it’s description and vice versa.