SimTK: SimTK Website: [#3018] [data share] Support for large datasets

Share
Follow	Project Follow publicly Follow privately Unfollow See followers (5) List Map More info

Trackers

Date:
2020-11-16 19:39

Priority:
4

State:
Closed

Submitted by:
Joy Ku (joyku)

Assigned to:
Henry Kwong (hykwong2012)

Target Release:
4.2.0

Status:
Testing complete

Summary:
[data share] Support for large datasets

Detailed description
- Implement dataset quotas - Develop solution for downloading large, publicly available datasets. Simple solution may be to notify user: Add message "This is a large dataset and will take a long time to retrieve. Do you want to continue?"
- Implement dataset quotas - Develop solution for downloading large, publicly available datasets. Simple solution may be to notify user: Add message "This is a large dataset and will take a long time to retrieve. Do you want to continue?"

Comments (9)
Attachments
Changes

Add A Comment:

Message
Date: 2022-11-17 22:02 Sender: Joy Ku Confirmed updates on Edge.
Date: 2022-05-17 08:31 Sender: Anthony Souza - tested on Firefox, chrome and safari and download is working as expected.
Date: 2022-05-17 00:46 Sender: Henry Kwong - The issues with "Cannot Process Request" and "Missing file parameters" have been fixed.
Date: 2022-05-13 06:34 Sender: Anthony Souza -- (tested and working) When data-set is too large a message is shown and an email is sent when archive ready for download. - (Issue) When clicking link in email, if user is logged out when link is clicked and then logs in, download fails with "Cannot Process Request" - (Issue) When not logged in, cannot download folder, download fails with "Missing file parameters" after agreeing to study license. Files larger than 200MB work fine by themselves.
Date: 2022-04-22 01:46 Sender: Henry Kwong - The email has been updated as described.
Date: 2022-01-18 07:17 Sender: Joy Ku I was able to download the file now. Let me do some more testing to verify whether this works consistently.
Date: 2022-01-18 06:47 Sender: Joy Ku - Update email text that lets user know the zip file is ready to the following: 1) Subject line: The data you requested from project "XXXX" is ready for download 2) Email content: Hi <NAME>, The data you requested from project "XXXX" is ready for download from this link. You may need to copy-and-paste the link into your browser. Note: if the data file does not download, make sure you have allowed pop-ups. In most browsers, you can do this by selecting the small icon to "allow pop-ups..." in the URL bar. Once pop-ups are allowed, click the link again to download the data file. The link will expire in 2 days. Best regards, The SimTK Team - When I click on the link in the email, I am unable to log in to access the file. - When someone who is not logged into SimTK tries to download a large set of data, the error message "Missing files parameter" is displayed. - The 10/22 comment mentions that the disk usage monitoring was implemented. Is that the disk quota referred to in this issue: https://simtk.org/tracker/index.php?func=detail&aid=3137&group_id=11&atid=1960? Or is this a different feature? If the latter, please provide some more details for testing purposes.
Date: 2021-10-22 21:59 Sender: Henry Kwong - For registered SimTK user, download of selection of directories or files in a large dataset that results in a large zip file is handled by cronjob. - Email is sent to the registered SimTK user when zip file generation is complete. - Disk usage is monitored.
Date: 2021-10-21 23:54 Sender: Henry Kwong - Created a new issue for quota checks across several modules which includes "Implement dataset quotas". Hence, for dataset quotas, refer to the issue below: https://simtk.org/tracker/index.php?func=detail&aid=3137&group_id=11&atid=1960

Attachments:

Size	Name	Date	By	Download
85 KiB	cannotprocess.jpg	2022-05-13 06:34	asouza88	cannotprocess.jpg
63 KiB	missing file parameters.jpg	2022-05-13 06:34	asouza88	missing file parameters.jpg

Field	Old Value	Date	By
status_id	Open	2022-11-21 21:53	hykwong2012
close_date	None	2022-11-21 21:53	hykwong2012
assigned_to	joyku	2022-11-17 22:02	joyku
Target Release	4.1.7	2022-10-27 19:10	hykwong2012
assigned_to	asouza88	2022-05-17 08:31	asouza88
Status	Ready for testing	2022-05-17 08:31	asouza88
assigned_to	hykwong2012	2022-05-17 00:46	hykwong2012
Status	Implementing	2022-05-17 00:46	hykwong2012
assigned_to	asouza88	2022-05-13 06:34	asouza88
Status	Testing	2022-05-13 06:34	asouza88
File Added	786: cannotprocess.jpg	2022-05-13 06:34	asouza88
File Added	787: missing file parameters.jpg	2022-05-13 06:34	asouza88
assigned_to	joyku	2022-05-11 22:31	hykwong2012
Target Release	4.1.6	2022-04-12 00:22	hykwong2012
assigned_to	hykwong2012	2022-01-18 07:17	joyku
Status	Implementing	2022-01-18 07:17	joyku
assigned_to	joyku	2022-01-18 06:48	joyku
Status	Ready for testing	2022-01-18 06:48	joyku
assigned_to	hykwong2012	2021-10-22 21:59	hykwong2012
Target Release	None	2021-10-22 21:59	hykwong2012
Status	Accepted	2021-10-22 21:59	hykwong2012