Data ingestion

Once you have crafted and polished your question to a degree to which you are satisfied with, it is now time to gather the raw data that will help you answer the question. This doesn't mean that your question cannot be changed once you go on to the next steps of the pipeline. You should continuously refine your problem statement and adjust it as necessary.

Collecting the right data for your pipeline might be a tremendous undertaking. Depending on the problem you are trying to solve, obtaining relevant datasets might be quite difficult.

Another important consideration is to decide how will the data be sourced, ingested, and stored:

• What data provider or vendor should we use? Can they be trusted?
• How will it be ingested? Hadoop, Impala, Spark, just Python, and so on?
• Should it be stored as a file or in a database?
• What type of database? Traditional RDBMS, NoSQL, graph.
• Should it even be stored? If we have a real-time feed into the...