您的位置:首頁>正文

技巧:Linux環境下載超大檔

前言

最近在學校的遠端深度學習伺服器上跑資料, 需要從境外伺服器上下載幾十G的訓練資料。

剛開始嘗試用curl下載, 下載到一半總是失去連接, 直到用上aria2和screen這兩個神器。

centos下面shell環境

top命令

aria2

安裝

# Ubuntusudo apt-get install aria2# Fredorayum install aria2# 手動編譯wget https://github.com/aria2/aria2/releases/download/release-1.31.0/aria2-1.31.0.tar.gztar xzvf aria2-1.31.0.tar.gzcd aria2-1.31.0./configuremakemake install

下載命令:

aria2c -s 5 -x 5 -c http://xxxxx# [-s 5] 檔分割為5段下載# [-x 5] 最大連接數5# [-c] 支援中斷點續傳# 選項-s, --split= Download a file using N connections. If more than N URIs are given, first N URIs are used and remaining URIs are used for backup. If less than N URIs are given, those URIs are used more than once so that N connections total are made simultaneously. The number of connections to the same host is restricted by the --max-connection-per-server option. See also the --min-split-size option. Default: 5 NOTE: Some Metalinks regulate the number of servers to connect. aria2 strictly respects them. This means that if Metalink defines the maxconnections attribute lower than N, then aria2 uses the value of this lower value instead of N.-c, --continue[=true|false] Continue downloading a partially downloaded file. Use this option to resume a download started by a web browser or another program which downloads files sequentially from the begin- ning. Currently this option is only applicable to HTTP(S)/FTP downloads.-x, --max-connection-per-server= The maximum number of connections to one server for each download. Default: 1screen

通過 ssh 登錄伺服器下載檔案, 但如果斷開登錄連接, aria2也會停止運行。

下面給出兩種方法解決這個問題:

方法1 nohup

nohup aria2c -s 5 -x 5 -c http://xxxxx > nohup.txt &# [nohup] 忽略 SIGHUP 信號# & 後臺運行程式

方法2 screen

screen 可以理解為一個虛擬螢幕;

在同一個終端下擁有多塊“螢幕”——即“會話”。

#創建一個離線會話screen -dmS test#查看會話列表screen -ls There is a screen on: 19393.test (Detached) 1 Socket in /var/run/screen/S-root.#進入會話screen -r test# 或者screen -r 19393# 執行下載命令aria2c -s 5 -x 10 -c http://xxxxx#退出會話C-a d#刪除會話screen -d test#下次進入會話重複上面的步驟即可#退出會話後, 裡面運行的程式不會停止

到此, linux環境下面下載大檔就再也不用愁了, 哈哈。

同類文章
Next Article
喜欢就按个赞吧!!!
点击关闭提示