Data Analysis with STATA

Chapter 1. Introduction to Stata and Data Analytics

These days, many people use Stata for econometric and medical research purposes, among other things. There are many people who use different packages, such as Statistical Package for the Social Sciences (SPSS) and EViews, Micro, RATS/CATS (used by time series experts), and R for Matlab/Guass/Fortan (used for hardcore analysis). One should know the usage of Stata and then apply it in one's relative fields. Stata is a command-driven language; there are over 500 different commands and menu options, and each has a particular syntax required to invoke any of the various options. Learning these commands is a time-consuming process, but it is not hard. At the end of each class, your do-file will contain all the commands that we have covered, but there is no way we will cover all of these commands in this short introductory course.

Stata is a combined statistical analytical tool that is intended for use by research scholars and analytics practitioners. Stata has many strengths, but we are going to talk about the most important one: managing, adjusting, and arranging large sets of data. Stata has many versions, and with every version, it keeps on improving; for example, in Stata versions 11 to 14, there are changes and progress in the computing speed, capabilities and functionalities, as well as flexible graphic capabilities. Over a period of time, Stata keeps on changing and updating the model as per users' suggestions. In short, the regression method is based on a nonstandard feature, which means that you can easily get help from the Web if another person has written a program that can be integrated with their software for the purpose of analysis. The following topics will be covered in this chapter:

Introducing Data analytics
Introducing the Stata interface and basic techniques

Introducing data analytics

We analyze data everyday for various reasons. To predict an event or forecast the key indicators, such as the revenue for a given organization, is fast becoming a major requirement in the industry. There are various types of techniques and tools that can be leveraged to analyze the data. Here are the techniques that will be covered in this book using Stata as a tool:

Stata programming and data management: Before predicting anything, we need to manage and massage the data in order to make it good enough to be something through which insights can be derived. The programming aspect helps in creating new variables to treat data in such a way that finding patterns in historical data or predicting the outcome of given event becomes much easier.
Data visualization: After the data preparation, we need to visualize the data for the the following:
- To view what patterns in the data look like
- To check whether there are any outliers in the data
- To understand the data better
- To draw preliminary insights from the data
Important statistical tests in Stata: After data visualization, based on observations, you can try to come up with various hypotheses about the data. We need to test these hypotheses on the datasets to check whether they are statistically significant and whether we can depend on and apply these hypotheses in future situations as well.
Linear regression in Stata: Once done with the hypothesis testing, there is always a business need to predict one of the variables, such as what the revenue of the financial organization will be in specific conditions, and so on. These predictions about continuous variables, such as revenue, the default amount on a credit card, and the number of items sold in a given store, come through linear regression. Linear regression is the most basic and widely used prediction methodology. We will go into details of linear regression in a later chapter.
Logistic regression in Stata: When you need to predict the outcome of a particular event along with the probability, logistic regression is the best and most acknowledged method by far. Predicting which team will win the match in football or cricket or predicting whether a customer will default on a loan payment can be decided through the probabilities given by logistic regression.
Survey analysis in Stata: Understanding the customer sentiment and consumer experience is one of the biggest requirements of the retail industry. The research industry also needs data about people's opinions in order to derive the effect of a certain event or the sentiments of the affected people. All of these can be achieved by conducting and analyzing survey datasets. Survey analysis can have various subtechniques, such as factor analysis, principle component analysis, panel data analysis, and so on.
Time series analysis in Stata: When you try to forecast a time-dependent variable with reasonable cyclic behavior of seasonality, time series analysis comes handy. There are many techniques of time series analysis, but we will talk about a couple of them: Autoregressive Integrated Moving Average (ARIMA) and Box Jenkins. Forecasting the amount of rainfall depending on the amount of rainfall in the past 5 years is a classic time series analysis problem.
Survival analysis in Stata: These days, lots of customers attrite from telecom plans, healthcare plans, and so on, and join the competitors. When you need to develop a churn model or attrition model to check who will attrite, survival analysis is the best model.

The Stata interface

Let's discuss the location and layout of Stata. It is very easy to locate Stata on a computer or laptop: after installing the software, go to the start menu, go to the search menu, and type Stata. You can find the path where the file is saved. This depends on which version has been installed. Another way to find Stata on the computer is through the quick launch button as well as through Start programs.

The preceding diagram represents the Stata layout. The four types of processors in Stata are multiprocessor (two or four), special edition processor (flavors), intercooled, and small processor. The multiprocessor is one of the most efficient processors. Though all processor versions function in a similar fashion, only the variables' repressors frequency increases with each new version. At present, Stata version 11 is in demand and is being used on various computers. It is a type of software that runs on commands. In the new versions of Stata, new ways, such as menus that can search Stata, have come in the market; however, typing a command is the simplest and quickest way to learn Stata. The more you use the functionality of typing the command, the better your understanding becomes. Through the typing technique, programming becomes easy and simple for analytics. Sometimes, it is difficult to find the exact syntax in commands; therefore, it is advisable that the menu command be used. Later on, you just copy the same command for further use. There are three ways to enter the commands, as follows:

Use the do-file program. This is a type of program in which one has to inform the computer (through a command) that it needs to use the do-file type.
Type the command manually.
Enter the command interactively; just click on the menu screen.

Though all the three types discussed in the preceding bullets are used, the do-file type is the most frequently used one. The reason is that for a bigger file, it is faster as compared to manual typing. Secondly, it can store the data and keep it in the same format in which it was stored. Suppose you make a mistake and want to rectify it; what would you do? In this case, the do-file is useful; one can correct it and run the program again. Generally, an interactive command is used to find out the problem and later on, a do-file is used to solve it. The following is an example of an interactive command:

Reading data in Stata

Whenever data is inserted in Stata, it's copied into the RAM memory of the computer. Generally, some of the changes are not on the permanent side and are not saved. So, these changes are lost when you reopen the Stata session. You can enter the data into Stata in various ways. One of the most effective way is as follows:

Use E:\Stata1\t1  less  India pwt  80-2010.dta,  clear

The option at the end of the code, clear, makes Stata read the dataset again before you open another data file.

Another option with limited variables in the dataset is as follows:

use  country  year  using  "t1  less India  pwt  80-2010 . dta" ,  clear

Insheet

In order to read data in Stata, it has to be converted into a format other than Excel. Also, save the data in one of the following formats:

Excel
CSV (comma separated values)
Text (where the delimiter is a tab or comma)

You need to take into consideration certain rules and regulations while working on Stata:

Suppose that the first row in the Excel file contains the name of the variables or headers, that is, the sheet contains variable names (series/code/names). Then, the second row must have data. The title of the first row must be removed before saving the file.
In Stata, every single word is read; therefore, any additional lines below or to the right of the data, for example, footnotes or endnotes, should be deleted before saving it. If essential, delete the entire bottom row or the column on the right-hand side.
You should not put numbers in the beginning of the variable name. In Stata, a problem might occur when the file is arranged with years (1980, 1985) in the top row. In such cases, placing an underscore before numbers will be helpful, and this can be done by selecting the row, using the spreadsheet package, and finding replace tools; for example, 1980 becomes _1980, and so on.
The most important thing to note is the deletion of commas from the data because Stata won't be able to understand the starting point and finishing point of columns and rows. You can do this by leveraging the first find then replace option.
Notations such as double dots (..) or hyphens (-) might trouble Stata and will create confusion because Stata can read a single dot (.) as double dots or hyphens as text.

After saving the data in the CSV format, it can be read in Stata, as shown in the following code snippet:

insheet using "E:\Stata1|t1 less India pwt 80-2010.  txt",  clear

If any changes are made to the data by applying the cd command, then it can be read as follows:

insheet using "t1 less India pwt 80-2010.  txt",  clear

Many ways are available for the insheet command. Options are defined as additional qualities of standard commands, which are generally added once the command ends, should have commas in between, and so on. The following are some of the options used in Stata:

The clear option: This can be used to insert a new file, insheet, regardless of the selected data: insheet using "E:\ Stata1\t1 less India pwt 80-2010 . txt" , clear
The option name: This provides insights of data (usually from the first row), which helps Stata remember the file automatically. However, in certain cases, if this option does not work, then Stata uses variable names; an example is as follows:
```
insheet using "E:\Stata1 classes\t1 less India pwt 80-2010 . txt" , names  clear
```
The delimiter option: This gives instructions to Stata regarding data insertion to insheet. Stata has the ability to recognize tab as well as comma-delimited data, yet often other delimiters such as ; are used in datasets. Here is an example:
```
insheet using "E:\Ind-samp.txt", delimiter (";")
```

Infix

Along with insheet, you can use the infix command, as shown later.

Most times, CSV or tab-delimited datasets are utilized, and the ASCII format is still used to save older data. Let's take the example of a survey taken by the government. This example represents two lines from 2010:

      10862226023331    06 022  3  02220155500666600777000003331
      10001222228332    06 022  3  02555553006666000000000044441

A codebook or data dictionary usually comes in the PDF or text file format. It explains the data that shows us that the first two numbers, the row ID, and the other two numericals are survey records (2010 from the previously mentioned dataset), and the fifth number is the quarter (the first quarter in this case) of the interview, among other things. infix is required to read such types of data and provides information to Stata from the codebook. The following is an example:

infix rowtype 1-2 yr 3-4 quart 5 […] using 
"E:\ Stata1\Survey2010.dat", clear

In order to save many files, the dictionary file is used; it will save the codebook information and mark it as a separate file. The file can be seen as follows:

infix dictionary using Survey2010.dat 
{
  dta
  rowtype  1-2 
  yr  3-4 quart5 […]
}

The infix command is used after saving the data as Survey2010.dct. As a relative path is used in the dictionary file (Survey2010), it is believed that raw data will be inside the same file set that is either a dictionary or a catalogue file. This being the case, then referring data is not required. The file will look like this:

infix using "H:\ECStata\NHIS1986.dct", clear

Defining and constituting a dictionary file in a proper way is a tedious job. However, NHIS has a dictionary that can be read through the SAS program; this can be converted into Stata using the Stat/Transfer program.

The Stat/Transfer program

This program is used to convert various dataset formats into well-defined industry formats, such as SAS, R, SPSS, Excel, and so on. Before converting, the data should be examined thoroughly. As it is an extremely user-friendly tool, it can be used to change the data between various packages as well as formats. This is shown as follows:

Manual typing or copy and paste

Typing or copying and pasting is the same as in other programs, but here, it can be done through the Stata editor. Just select the required data columns in Excel and paste them in the Stata editor. However, this has some drawbacks; many times, data inaccuracy or missing values don't have any fixed procedure, and in certain cases, language problems may arise. For example, in selected countries, a comma is used instead of a decimal point.

Typing is an extremely tough job, especially when electronic data is unavailable because in that case, we have to type the data. This job becomes easy in Stata through the edit command as it will take you to a spreadsheet-like feature where new data can be entered and old data can be edited.

Variables and data types

There are different types of variables and data types, which we are going to see in this section.

Indicators or data variables

To find the insights and the data conclusions, the browse/edit command is helpful. Data variables store the fundamental data. As shown in the following table, the income data for different nations is stored in the Cccgdp variable and the country (Countrycode) data is stored in the pop variable. If we want to get an idea about the details of all kinds of data, then one indicator variable is needed. In the following case, Countrycode and yr will provide information regarding the country, the year, the country's GDP, and the population data (pops). The data might be as follows:

Country	Countrycode	Yr	Pops	Cccgdp	Openss
India	IND	2010	23452.9	10897.23	23.11111
U.S.	USA	2010	22222.1	23987.23	90.42231
Pakistan	PAK	2010	11111.2	23675.21	10.22291
China	CHN	2010	98765	97654.94	30.98765
Russia	RUS	2010	19876	65745.11	43.34343
Germany	GER	2010	23467	23874.35	23.74747

After importing the data in Stata, it is always a good practice to examine the data. It gives you an advantage in any modeling or visualization exercise.

Examining the data

Examining the data is always recommended. It is a good idea to examine your data when you first read it into Stata; you should check whether all the variables and observations are present and are in the correct format.

While the browse/edit command is used to examine the raw data, the list command is used to see the results of the data. Listing small data is possible through this command. For bigger datasets, options are used to track the data. An example is shown as follows:

List country* yr pops

Country       countrycode     yr        pops 
India         IND             2010      23452.9 |
U.S.          USA             2010      22222.1 |
Pakistan      PAK             2010      11111.2 |
China         CHN             2010      98765 |
Russia        RUS             2010      19876 |
Germany       GER             2010      23467 |

In the preceding table, the star is called the placeholder, and it instructs Stata to incorporate the entire data with the country. Alternatively, we could focus on all variables but list only a limited number of observations, for example, the observation from 14th to 19th row:

The following table contains the country, country code, year, and pops 14/19:

Country	Countrycode	Yr	Popscon	Cccgdps	kOpenss
India	IND	2010	23452.9	10897.23	23.11111
U.S.	USA	2010	22222.1	23987.23	90.42231
Pakistan	PAK	2010	11111.2	23675.21	10.22291
China	CHN	2010	98765	97654.94	30.98765
Russia	RUS	2010	19876	65745.11	43.34343
Germany	GER	2010	23467	23874.35	23.74747

How to subset the data file using IN and IF

In the previous part, the in qualifier was used; it makes sure that the subset pertains to selected data. A lot of observations follow after this, for example:

The list in 14/19
The list in 90/l
The list in 30/l

As is clear from the preceding example, there are three observations:

The first command lists observations from 14 to 19
The second command lists 90 observations
The third command lists observations from 30 till the last observation

The if statement is the other way of subsetting data; it generally has values of true or false. The following is an example from the observation of the year 2010, where the variable name is yr:

list if yr == 2010

In order to examine the raw data, the browse window is used. However, a problem occurs when only selected variables are to be viewed; this happens in big datasets. So, in this condition, create a list of the variables you want to examine before browsing. This is done through the following command:

browse country yr popscon

It is important to note that this edit command will help change the dataset manually. The assert command helps Stata examine the observation. This is because when the bigger data (or big data, as it is called in today's world) arrives, checking single data through browse or edit commands becomes difficult. In this case, the assert command is helpful. There are a couple of advantages: it helps identify whether a data statement is right or wrong. For example, in the case of the population of the country (popscon), it will tell us that the values are positive:

assert popscon>0,
assert popscon<0

If the preceding command results in the value true, then assert does not give any output. However, if the command value is false, then an error message will appear.

The describe command accounts for various fundamental information regarding datasets and variables, such as the total size of the dataset and the variable, the total number of variables in the dataset, and different formats of the variables. This can be denominated as describe. It can only be applied to an unread file in Stata. An example is given as follows:

describe using "E:\Ind-Health-sample.dta"

Codebook can give information on variables in the dataset without the list of variables; an example of this is codebook country.

The summarize command delivers the statistics summary: means, standard deviation, and so on. The following table represents this tab:

summarize table
Variable         Obs      Mean       Std. Dev.    Min         Max

Cntry	0
countrycode	0
Yr	97	2000	2.156	1990	2010
Popscon	97	87634.46	8374.33	29383.9	93830
ccCgdps	97	67544.23	4100.682	15890.71	98739.67
kOpenss	97	34	4	13	50
Chi-ppl	97	23.6	3.56	10.456	40.8796
Fdhsa	97	19.56	9.567	12.456	34.98765
Gdkliyu	97	1.987456	1.2	-3.238917	6.46896

As we can see in the preceding table, string variables such as Cntry and Countrycode do not have numbers; this is why no summary details are available. Yr is a numeric variable; therefore, we can see that it has a statistics summary. For more details, the summarize detail option can be used.

The wide range of graphic qualities makes Stata a unique tool. One can easily get help by typing the help command in Stata. A histogram graph can be created through the following command:

graph twoway histogram cccgdps

For a scatter plot, you have to leverage the following command:

graph two-way scatter ccccgdps popscon

Even though there is some benefit of having advanced graphs in Stata, this makes it work slowly. In certain cases, it is better to use version 7 graphics because they help visualize the data properly without using papers or presentations. This can be seen as follows:

graph7 cccgdps popscon

Saving the dataset is a very easy command, and it is represented as follows:

Save "E:\Stata1\t1 less India pwt 80-2010.dta", replace

If we have sets of files of the same content, then the replace tab/option can be helpful. It will swap the last version and save it. If the old version is to be stored for some reason, then save it with a different name. One thing that should be kept in mind is that the original file content can be changed if it is saved with revised datasets. Therefore, after changes are made to the revised file, in order to open the file and restart it, just reopen it.

There are two ways to preserve and store the data. One option is to save the current data and revise it, and later, if you don't want to keep the data, then reopen the saved data version. Another option is to use the preserve and restore functions/commands; they will take an image of the data, and the data will come back after you type restore.

Sampada May 30, 2018

As a strategy consultant, this book helped a lot to understand analytics concepts in simple language. The book is very good in explaining tough analytics methods in simple language. I have tried many ways to learn analytics and could not get beyond jargons. I really liked this book because of simple, straightforward language with good real life examples.

Amazon Verified review

SD Nov 10, 2018

Really liked the simple language and easy examples used to explain some complex analytics concepts. I am using Stata on healthcare and genome data and this book helped me a lot.

Alexander Huelle Sep 02, 2016

Da alle Titel, die ich bisher von Packt Publishing gelesen hatte, sehr gut waren, bestellte ich auch dieses Buch. Leider fehlt es ihm an so ziemlich allem, was ein gutes Buch ausmacht:- es ist ausgesprochen schlampig lektoriert. Nicht nur sind viele Schreibfehler stehen geblieben, bei einem Bild wurde sogar die Dummy-Darstellung nicht ersetzt.- die Darstellung der Themen enthält deutliche Lücken und Sprünge, die das Verständnis sehr erschweren- die Auswahl der Themen erscheint willkürlich und auch für einen Überblick zu fragmentarisch.Insgesamt: nicht empfehlenswert!.

Amazon Customer Jan 25, 2017

you're better off using another book as this one is too primitive and not helpful at all.

Christopher Bratt Apr 01, 2016

Not a very detailed book. But much more important, I've never before seen a book with so many typos. Some of them will make learning Stata rather difficult.Two examples:1."A Stata macro is not [sic] a black box where we can input [sic] the text and numbers."2.After using a correct template for foreach loops (with curly brackets), the text goes on to consistently using square brackets for denoting the beginning and the end of the loop.Added later:I looked throught the rest of the book. It's a complete waste of your money and time. Even if you disregard the many typos (assuming they were corrected): You will learn very little about Stata reading this book.

Data Analysis with STATA: Explore the big data field and learn how to perform data analytics and predictive modelling in STATA

What do you get with eBook?

Data Analysis with STATA

Chapter 1. Introduction to Stata and Data Analytics

Introducing data analytics

The Stata interface

Data-storing techniques in Stata

Directories and folders in Stata

Reading data in Stata

Insheet

Infix

The Stat/Transfer program

Manual typing or copy and paste

Variables and data types

Indicators or data variables

Examining the data

How to subset the data file using IN and IF

Summary

Page 1 of 8

Description

Who is this book for?

What you will learn

Product Details

What do you get with eBook?

Product Details

Frequently bought together

Table of Contents

Recommendations for you

Customer reviews

People who bought this also bought

About the author

FAQs

Data Analysis with STATA: Explore the big data field and learn how to perform data analytics and predictive modelling in STATA

What do you get with eBook?

Manual typing or copy and paste

How to subset the data file using IN and IF

Description

Who is this book for?

What you will learn

Product Details

What do you get with eBook?

Product Details

Packt Subscriptions

Frequently bought together

Table of Contents

Recommendations for you

Customer reviews

People who bought this also bought

About the author

FAQs