Article content
Jan Wiśniewski
8 min read
Data is arguably the most important component of the modern web – and data collection can help your business build better products and gain a deeper understanding of your user base. In this article, we’re analyzing the ins and outs of this process: Which data collection equipment can we use? Which data collection problems and challenges can we encounter? Which data collection methods should we choose in the given scenario?
What is Data Collection?
Data collection is aggregating numerous data points from various sources to transform it into actionable data later. This definition has a few important concepts – let’s unpack them:
Aggregating: Data can be collected via different methods. In some cases, this process is automated: Web scraping, for instance, involves running instances of data collection software that scrape web pages for all sorts of content: tables, images, videos, and more. In other scenarios, the data collection procedures are manual.
Data points: These are unique pieces of information. Together, they can form a complete picture of the given situation: Charts, for example, group individual data points (e.g. graphics card prices) to explain a concept or highlight an idea (e.g. the sharp decline of the price of said cards throughout 2022.)
Various sources: We can use different channels to find desirable data. These may include reports, statements, company information, interviews, surveys, and many more. Some sources are easily available to the general public, while others are less open; choosing the right source is an important step during data collection considerations.
Actionable data: The web and other sources are rich with large volumes of information – but it’s often not ready to generate value for you straight away. Raw web data has all sorts of unnecessary elements (HTML tags, irrelevant data points, and more) that make it hard to read and understand.
Actionable data, on the other hand, allows you to make informed decisions, create strategies, perform further research, and more: Reading it helps you get an insight into the given situation and understand it better. One important step in the overall data collection pipeline is data parsing, which removes irrelevant pieces from the raw data and makes it human-readable.
Types of Data Collection
The data collection pipeline uses two major types of information: quantitative data and qualitative data. As their names suggest, the distinction between the two data types lies in numbers and calculations vs. inherent categories.
Quantitative data deals with numeric variables. At its core, this data type answers questions like “How many?”, “How much?”, or “To what degree?”.
Qualitative data involves more abstract components like descriptions and opinions. At its core, this data type answers questions like “What type is it?” and “What is it?” Thus, this type of data may be hard to put into a formula and measure precisely, but some data is simply organized that way.
The nature of data influences its collection type, allowing you to predict whether the given information is quantitative or qualitative. In many scenarios, however, qualitative and quantitative data are used side by side to paint a broader picture or explain an idea in a clearer way. In a government census, for instance, information about annual income (e.g. $100,000) may be supplemented with occupation data (e.g. software engineer) to show correlations or discrepancies.
Let’s list some examples of quantitative and qualitative data:
| Data unit | Variable | Data type |
|---|---|---|
| Employee | Annual income | $100,000 (=quantitative) |
| Employee | Occupation | Software engineer (=qualitative) |
| Individual | Number of children | 2 (=quantitative) |
| Individual | Marital status | Married (=qualitative) |
| Company | Number of employees | 328 (=quantitative) |
| Company | Industry | Software development (=qualitative) |
Primary Data Collection Methods
The data collection process has two different methods, each offering its own strengths and weaknesses. Gathering data efficiently can be tricky – but choosing the right method can make a huge difference. Let’s take a closer look at popular methods of data collection: primary and secondary.
Primary data is information collected directly from first parties. In any area and industry, there are people holding individual data points – and if we aggregate their knowledge, experience, and expertise, we get data in the collective sense of the word. There are different primary data collection methods at our disposal for this job: interviews, questionnaires, observation, and more.
In-person interviews
Data collection performed in person (“offline”) is arguably the most powerful method: In addition to the responses themselves, the interviewer can also pick up on a wide range of verbal and non-verbal cues, which typically include voice cadence, posture, eye movement, and emotions – working together, these markers produce high data quality. This method is optimal for local businesses that want to obtain customer data and engage their clients to share their opinions about the quality of service.
Online interviews
The biggest advantage of online data collection methods is scalability: Unlike their offline counterparts, online surveys and interviews can be organized en masse; channels like social media and email newsletters can be used to invite customers to participate in said surveys. With the rising popularity of telecommunication,
Another advantage has to do with the digital data format: Online methods, by definition, store resulting information digitally. Particularly in the case of big data, digital format enables accurate data collection, helping companies save time and resources by processing collected information more quickly and efficiently. Additionally, this improves data integrity and prevents human mistakes during information processing.
Moreover, online surveys are easier to carry out anonymously, with many online survey platforms offering a secure and private environment. For this reason, this method is optimal for gathering service and product feedback.
Phone interviews
Phone surveys offer a blend of offline and online methods: They retain a degree of personal interaction – and offer a similar degree of privacy for those respondents who only want to reveal their voice. The main challenge involving phone data collection methods has to do with your audience’s responsiveness, which determines whether you’re making a cold or a warm call.
Each call type involves different strategies: Naturally, warm calling is much more effective as your audience is prepared for participation. Still, collecting data via phone interviews is a skill in and of itself, with sales representatives taking years to hone what types of questions they should ask first to keep their conversation partner interested. Phone interviews are best suited for gauging political interests, performing sociological studies, and more.
Mail surveys
Mail surveys may not seem like a common data collection method: Isn’t email the better choice in any scenario? However, like phone interviews, traditional mail is a great communication channel for senior respondents, who may be skeptical of online services and tools – or simply unable to use them. Moreover, mail surveys can be designed and drafted to motivate prospective respondents, promising coupons and special offers.
Secondary Data Collection Methods
Secondary data, conversely, comes from sources other than the researcher: It can be accessed from government archives and records, open-source databases, and more. Thus, you don’t need specialized methods to obtain secondary data: It may be stored across numerous copies in a wide range of sources. Secondary data collection methods involve two types of sources: internal and external.
Internal sources are marked by their relation with the given group: They typically include company reports, financial statements, employee information, and more. Thus, data privacy is a major contributing factor, increasing the difficulty of dealing with internal sources if you want to gather data. In some scenarios, information from this type of source will be much harder to acquire as some reports and statements are either incomplete or unavailable to the general public.
External sources, on the other hand, provide information that is readily available without major roadblocks: These sources may include open-source databases, public records, academic reports, and publicly available web data.
Primary Data Collection Examples
Structured and unstructured interviews
This format allows the interviewer to collect data from a group of respondents. A structured interview is more formal and typically has a set of choices for each question; an unstructured interview offers more flexibility, letting interviewees give answers to open-ended questions without prompts or tips.
The Delphi method
Delphi, an ancient Greek oracle, helped various figures make important political decisions. The Delphi method is a modern implementation of this practice: An expert (or a group of experts) in the given area answers the interviewer’s questions, which are used to come to a conclusion and make a decision.
The projective method
Combining an unstructured interview with abstract questions, this method, as the name suggests, forces the interviewee to project their thoughts. This way, they have to answer independently, which may reveal their true opinion on the given topic. In data collection, a subset of the projective method is the sentence completion method, which makes the respondent fill in the blanks in an incomplete sentence.
Word association tests
Similar to the projection method, word association works with the participant’s underlying feelings and standpoints: They’re given a set of words and asked what kind of associations arise when they hear the given word. This method is effective in data collection for marketing and brand campaigns in particular, helping the company gauge if the given name or tagline are catchy and customer-friendly.
Focus group interviews
An important subset of primary data collection via interviews involves inviting a small (5-10) number of respondents to talk about a specific topic or problem. For best results, the respondents need a motivation for discussing the given topic: This way, the interviewer can glean feedback with the most value. A good example of a focus group interview is gathering parents to discuss a child care program.
Role playing
In a role playing scenario, the respondents are placed in an imaginary situation to act it out and solve various problems. Although the given situation is make-believe, the participants are still acting according to their fundamental principles and motivations – this creates a data collection process where the interviewer can probe into their underlying opinions.
Secondary Data Collection Examples
The nature of secondary data influences the data collection toolset we can use: We’re essentially working with information provided by third parties, so some of it may not be as accurate. On other hand, collecting secondary data is cheaper and easier to scale – let’s see which sources are available to us:
Quantitative methods of secondary data collection include:
- Open-source databases with easy access (e.g. government and population censuses),
- Company data (annual reports, financial statements),
- Customer data (name, age, emails, etc.),
- And more.
Qualitative methods of secondary data collection include:
- Interviews and reports,
- Transcripts,
- School and police records,
- Media (movies, radio broadcasts, YouTube videos, social media posts),
- And more.
Frequently Asked Questions
The most common problem tied to data collection is arguably web scraping roadblocks: reCAPTCHA, Cloudflare, and similar anti-scraping systems. Major tech platforms (Amazon, Twitter, Google, etc.) use these systems to protect their data and prevent automated access to it.
Thankfully, this problem can be solved with residential proxies, which are the optimal tool for bypassing these restrictions.
Your toolset will largely depend on the type of data collection you perform – for instance, primary vs. secondary. You could start with choosing a data parsing library, which can help you acquire cleaner information that is easier to read – and easier to process at the later stages.
