ICT705 Data and System Integration Proof Reading Service

ICT705 Data and System Integration Proof Reading Service

1.Introduction

The aim of this report is to make Mashup Software, which will get the location of the clinics located in the various regions of the country. These Clinics are owned by the same organization. This is achieved in three phases:

1.Use of petl module to extract and load data from csv files to table and to convert XML file into the table which is later combined. Also, regex is used to clean the missing data from the CSV file.

2.In next step a RESTful WS is designed which is based on the Python Bottle Framework, it returns data in JSON format

3.In the end, a Mashup is developed which gives the GUI to the user, where a user could enter postcode of the known clinic state an get to know the location of the clinic.

To achieve this, IDE PyCharm is used and the code is developed using Python, HTML, CSS, and JS. 

2.Data Merging and Cleaning

The data given by the company has missing values, which in turn will lead to the inconsistent results. Also, the data given is in different formats which are merged later into one.To solve this issue

Files that needed to be analysed for this project:

1.Clinics.csv:This is a Comma Separated value file. In it, fictional locations of the clinics are given which are owned by the company in the different Suburbs. Figure 1 shows the data in the clinics.csv file.

Figure 1: Data in clinics.csv file

2.Location.xml: another file is an XML file in which contains Longitude and Latitude of Clinics in XML file. As it is independent of the platform so, no need to embed it with any other format. The below figure shows the XML data in the browser.

Figure 2: Data in location.xml file

Implemented method to convert XML to CSV

Figure 3: petl framework that converts the XML to CSV

Data Cleaning:

As said earlier, the clinic data that is given in the form of CSV file (Comma-Delimited Values) have missing values. The Email Values are missing which have to clean before the Merging process. To make those Emails understandable Regular Expression is used, in Python Regex module is used which in turn complete the data. The figure below shows the missing data (displayed by Red arrows).

Figure 4: Missing Data

Figure 5: Data after Cleaning

Code Implementation for Cleaning process

Figure 6: Code to clean the csv file

Data Merging

In this project of system design, we have two kinds of Data. To use them as one they need to be processed as together. In this process, those two files are merged together as one named under “clinic_locations.csv”.In this process, both files are converted into Tables using the petl module in the python and then merged together into one file.

Figure 7: Data after Merging

Code Implementation for Merging process

Figure 8: Code for merging process

Whole code for Cleaning and Merging

Figure 9: Code Snippet 

3.Implementation of RESTful Web Service

RESTful Web Service is implementedon web service architecture implemented by using a Virtual structure in a Hierarchical architecture. In this project, it is implemented by using BottleFramework. Bottle is a lightweight web (micro) framework used by python because it is simple to implement and the processing is fast. Bottle is used here because it doesn’t have any dependencies over any other Python library.

How to access RESTful Web Service

In this project, the URL used is

http://localhost:8080/getlocation?postcode=abcd (abcd is the postcode that the user will enter in the web browser to run this service)

Figure 10: Output when you run (http://localhost:8080/getlocation?postcode=2000)

To start the service there is a file called clinic_locator.py open that file in PyCharm and run it. This bottle implemented framework which will be running on default web browser. The data will be taken from the previously created csv file “clinic_locations.csv”. This web service will return the output in the format of JSON, which will be fetched later by the next process (discussed in next step). To run this process the system must have previously installed Bottle module in it.

Output in the PyCharm

Figure 11: clinic_locator.py in PyCharm

Implemented Code for this RESTful Web Service

Figure 12: Code to run RESTful Web Service (clinic_locator.py)

Output of this phase

Figure 13: RESTful Web Service Output

The Figure 12 shows the output that we get after running it in PyCharm. The output is in JSON format because it will send the data over network in a structured way. Also, JSON is the most used format to communicate between the servers sometimes web applications.

Output Explanation

The in above figure the postcode that is passed is 2000 which is postcode of Sydney.

The output has several values

1.Clinic Name: Sydney Centre Clinic

2.State: NSW

3.Email: Sydney@myclinic.com.au

4.Longitude: 151.207114

5.Latitude: -33.867139 

4.Mashups

Mashups is a hybrid application that is developed from two or more sources like web application or web pages. Mashups use those applications to create a single application or in a single graphical interface. Like in this project 

1.A RESTful Web Service is made which takes input from the output of the python code written of the on system processing of the data

2.The Output of the RESTful Web Service is fed as an input to the Mashup application as JSON data and further processing is done.

In this,  HTML file is created with the help of CSS and JavaScript from scratch. The file created in this process is clinic_map.html. The Objective of this Mashup is:

1.The Created HTML web page will take input from user in the form of postcode

2.The HTML page will fetch the data from the RESTful web service created in the previous step in the JSON form and will display the location on the Google map.

For showing the Google map the Google API is used. From this API I could show the location of the clinic whose postcode is entered in the search bar.

Sources of this Mashup

There are two sources of this Mashup:

1.The clinic_locations.csv file which I have created in the first step after Data Cleaning and Data Merging processes on the Locations.xml and Clinics.csv files.

2.The RESTful Web Service That is performed in the previous step in order to start the Web Service.

Implemented code for this process

There are three programming techniques used:

HTML Code

Figure 14: HTML code to implement the clinic_map.html

JavaScript Code

Figure 15: JavaScript Snippet One

Figure 16: JavaScript snippet Two

CSS Code

Figure 17: Index.css

Figure 18: Style file in the CSS

Output

Run Clinic_map.html in the browser

Figure 19: Home Screen

Figure 20: Entry Field

Figure 21: Mandatory Field

Figure 22: Server Not running Error

Figure 23: Output showing location on the map

The submit query button will directly redirect to the mail client and the subject line will already be filled with the “Clinic Enquiry”.

5.Demo Running Instructions

The project is implemented in three Phases:

1.Data Cleaning and Merging

2.Making RESTful Web Service using Python Bottle

3.Creating Mashup using above two phases

Perquisites:

1.Petl framework,

2.Pandas,

3.Bottle must be installed

For Data Cleaning and Data Merging

First, open PyCharm IDE in it

1.Open file named data_merger.py and hit configure and run

Figure 24: Click on Run

Figure 25: Click on Configurations

2.Find the specified path of the Python Interpreter

Figure 26: Configure the Python path

3.Run the Python code

Figure 27: Output

For running RESTful Web Service

The process is same for this service also.

Just Edit the path and run the application

Figure 28: Select clinic_locatior.py from the run option

Figure 29: Running the Service

For running RESTful Web Service

1.To run this the user must have an internet connection for the Google maps to fetch.

2.Now click on clinic_map.html file which will open it in your default web browser

Figure 30: Click on clinic_map.html file

1.Loading Page will be displayed

Figure 31: Open in the default browser

1.You will see the figure shown below

Figure 32: Homepage of the Mashup service

1.Enter the postcode of the clinic to find the location of the clinic

Conclusion

In the Nutshell, this report focuses on creating a Mashup application using which the patients can find the location of the desired clinic using the known postcode of that clinic. The postcode will search in the csv and match up with the location if the location is displayed it will be shown if not present the default location will be shown.

To achieve this, several things are done. At first, the data that is given is cleaned for missing values and then merged with the different file types. The output is then stored in a file which is used by both clinic_locator.py and clinic_map.html as an input file. This is an important step as it will change the entire result.

In the second step, a web server is designed using Bottle framework. It takes clinic_locations.csv as input and returns the JSON data which is later used by clinic_map.html to search and display the locations.

In the last step, a Mashup application is developed using above two processes. Using the files created above as input this Mashup application displays the location of the clinic by getting the postcode from the user. Also, an Enquiry button is given which will help the user to send a mail with an already filled subject line "Clinic Enquiry” in it.