Data Version Control VS Git

Git: (Manages source code)

• Stores source code, configuration files etc

• Version controls

• Provides auditing feature.

• Provides Role based access control

Limitations:

• Git is not designed for Large data(GB,TB)

• Slows down when multiple people handles(push/pull) large files

• Cost Ineffective when it comes to large files

DVC: (Manages datasets)

• Stores Data sets that are required to train the models

• Supports Large data sets.

• Comes with version control capabilities

• Cost effective

• Durable

How to use DVC+Object store+GIT for managing data sets.

#Installation
pip install git
pip install dvc
pip install dvc-s3

#Initialization
git init
dvc init
dvc remote add -d NAME URI://NAME_OF_BUCKET 

#Adds the datasets to Objectstore
#Adds the metadata to GIT
dvc add sample.csv      # creats sample.csv.dvc 
git add sample.csv.dvc  # file with metadata and checksum           
git add .dvc            # .dvc/config specifies the location of Objectstore
git commit -m "storing dataset metadata and objectstore location in Git"

#push the changes
git push      # pushes metadata and objectstore information to GIT 
dvc push.     # pushes the data set to Objectstore

GIT metadata information is the source of truth for the latest datasets being used.