XetHub

RedPajama-Data-1T

Assembled from URLs hosted at https://huggingface.co/datasets/togethercomputer/RedPajama-Data-1T

https://together.xyz/blog/redpajama

Fix up Makefile and check sha256 checksums #1

Merged

zach merged 2 commits from :main into main

zach commented

Owner

The Makefile is now usable. Running make will download all files in parallel (with up to 6 simultaneous requests with curl), then check all sha256 checksums.

jgodlew was assigned by zach

rajatarya was assigned by zach

saba was assigned by zach

seanses was assigned by zach

zach added 2 commits

5556539c40 Fix up Makefile and check sha256sums

6b500ca4e6 Switch to curl --parallel

saba approved these changes

jgodlew was unassigned by zach

rajatarya was unassigned by zach

saba was unassigned by zach

seanses was unassigned by zach

zach merged commit 46dca54338 into main 1 year ago

zach referenced this issue from a commit

Merge pull request 'Fix up Makefile and check sha256 checksums' (#1) from zach/RedPajama-Data-1T:main into main

Reviewers

saba approved these changes

The pull request has been merged as 46dca54338.

No Label

No Milestone

No Assignees

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: XetHub/RedPajama-Data-1T#1