large scale data management

Page rank in Pig, based on https://gist.github.com/jwills/2047314 Modified for running on Google Cloud Dataproc.

data

Data are supposed to be upload fist on Google Cloud Storage.

We need to create a root bucket to store you data and code

gcloud storage buckets create gs://BUCKET_NAME --project=PROJECT_ID  --location=europe-west1 --uniform-bucket-level-access

Next you can copy data/code using gstutils (see run.sh)

bzcat page_links_en.nt.bz2 | wc -l
 119077682

Ensure data and code are uploaded. Next update dataproc.py accordingly then run.

Do not forget to stop your cluster at when job is finished.

hints: 1 master / 2 slaves on page_links

2022-09-22 17:10:37,648 [main] INFO  org.apache.pig.Main - Pig script completed in 1 hour, 16 minutes, 49 seconds and 944 milliseconds (4609944 ms)

Name		Name	Last commit message	Last commit date
Latest commit History 60 Commits
Wikipedia-Stream		Wikipedia-Stream
data		data
pyspark		pyspark
CharSets.ipynb		CharSets.ipynb
GraphFramesPageRank.ipynb		GraphFramesPageRank.ipynb
PageRank_an_PySPARK_SQL.ipynb		PageRank_an_PySPARK_SQL.ipynb
PyPageRank.ipynb		PyPageRank.ipynb
README.md		README.md
Spark_Structured_Streaming.ipynb		Spark_Structured_Streaming.ipynb
Streaming_CSV.ipynb		Streaming_CSV.ipynb
VoiD_with_PySpark_RDD.ipynb		VoiD_with_PySpark_RDD.ipynb
Wikipedia_Events.ipynb		Wikipedia_Events.ipynb
authors.pig		authors.pig
dataproc.py		dataproc.py
exam2022.ipynb		exam2022.ipynb
flatten2.pig		flatten2.pig
map_reduce_with_py_sparl.ipynb		map_reduce_with_py_sparl.ipynb
people.csv		people.csv
run.sh		run.sh
small_page_links.nt		small_page_links.nt
stream.py		stream.py
vianu.pig		vianu.pig
watdiv-100k.nt		watdiv-100k.nt
webdam-books.txt		webdam-books.txt
webdam-publishers.txt		webdam-publishers.txt
wordcount.sh		wordcount.sh