Rcrawler::Crawling

Published by onesixx on 20-08-0720-08-07

Rcrawler로 crawling와 scraping 둘다 가능하다.

https://www.pluralsight.com/guides/web-crawling-in-r

R을 이용한 퀀트 투자 포트폴리오 만들기 – Chapter 4 크롤링 이해하기

특정 웹사이트 크롤링하기

CSS나 Xpath를 사용해서 URLs 찾기

### Title: ~~~
source(file.path(getwd(),"../00.global_quant.R"))
# > system("java -version")
# #java version "1.8.0_202"
# #Java(TM) SE Runtime Environment (build 1.8.0_202-b08)
# #Java HotSpot(TM) 64-Bit Server VM (build 25.202-b08, mixed mode)
library(Rcrawler)  # require Java 
myURL="https://finance.naver.com/news/news_list.nhn?mode=LSS2D&section_id=101&section_id2=258"

Rcrawler(Website=myURL,	 no_cores=4, no_conn=4, Obeyrobots=T,  DIR=DATA_PATH)
###### finance.naver.com-091606 폴더가 만들어 진다. 

# Preparing multihreading cluster .. In process : 1..
# Progress: 100.00 %  :  1  parssed from  1  | Collected pages: 1  | Level: 1 
# + Check INDEX dataframe variable to see crawling details 
# + Collected web pages are stored in Project folder 
# + Project folder name : finance.naver.com-091606 
# + Project folder path : ~/DATA/quant/cData/finance.naver.com-091606

page <- LinkExtractor(url=myURL, ExternalLInks=T) 
> page
# $Info
# $Info$Id
# [1] 638

# $Info$Url
# [1] "https://finance.naver.com/news/news_list.nhn?mode=LSS2D&section_id=101&section_id2=258"

# $Info$Crawl_status
# [1] "finished"

# $Info$Crawl_level
# [1] 1

# $Info$SumLinks
# [1] 95

# $Info[[6]]
# [1] ""

# $Info$Status_code
# [1] 200

# $Info$Content_type
# [1] "text/html"

# $Info$Encoding
# [1] "EUC-KR"

# $Info$Source_page
# [1] "\n\n\n\n\n\n\n\n<!--  global include -->\n\n\t\n\t\n\t\n\t\n<html lang='ko'>\n<...showPannel(layerId);\n        }\n    }\n}\n\n// add data-useragent\ndocument.documentElement.setAttribute('data-useragent',navigator.userAgent);\n</script>\n</div>\n</body>\n</html>"

# $Info$Title
# [1] "³×ÀÌ¹ö ±ÝÀ¶"

# $InternalLinks
#  [1] "https://finance.naver.com/news/news_list.nhn?mode=LSS2D&section_id=101&section_id2=258/"                                                                                                            
#  [2] "https://finance.naver.com/"                                                                                                                                                                         
#  [3] "https://finance.naver.com/sise/"                                                                                                                                                                    
#  [4] "https://finance.naver.com/world/"                                                                                                                                                                   
#  [5] "https://finance.naver.com/marketindex/"                                                                                                                                                             
#  [6] "https://finance.naver.com/fund/"                                                                                                                                                                    
#  [7] "https://finance.naver.com/research/"                                                                                                                                                                
#  [8] "https://finance.naver.com/news/"                                                                                                                                                                    
#  [9] "https://finance.naver.com/mystock/"                                                                                                                                                                 
# [10] "https://finance.naver.com/news/news_list.nhn?mode=LSS2D&section_id=101&section_id2=258"                                                                                                     
# [11] "https://finance.naver.com/news/mainnews.nhn"                                                                                                                                                        
# [12] "https://finance.naver.com/news/news_list.nhn?mode=LSS3D&section_id=101&section_id2=258&section_id3=401"                                                                                 
# [13] "https://finance.naver.com/news/news_list.nhn?mode=LSS3D&section_id=101&section_id2=258&section_id3=402"                                                                                 
# [14] "https://finance.naver.com/news/news_list.nhn?mode=LSS3D&section_id=101&section_id2=258&section_id3=403"                                                                                 
# [15] "https://finance.naver.com/news/news_list.nhn?mode=LSS3D&section_id=101&section_id2=258&section_id3=404"                                                                                 
# [16] "https://finance.naver.com/news/news_list.nhn?mode=LSS3D&section_id=101&section_id2=258&section_id3=406"                                                                                 
# [17] "https://finance.naver.com/news/news_list.nhn?mode=LSS3D&section_id=101&section_id2=258&section_id3=429"                                                                                 
# [18] "https://finance.naver.com/news/news_list.nhn?mode=RANK"                                                                                                                                             
# [19] "https://finance.naver.com/news/news_list.nhn?mode=LSTD&section_id=101&section_id2=258&type=1"                                                                                           
# [20] "https://finance.naver.com/news/news_list.nhn?mode=TV&section_id=tv"                                                                                                                             
# [21] "https://finance.naver.com/news/market_notice.nhn"                                                                                                                                                   
# [22] "https://finance.naver.com/news/market_special.nhn"                                                                                                                                                  
# [23] "https://finance.naver.com/news/news_search.nhn"                                                                                                                                                     
# [24] "https://finance.naver.com/news/news_search.nhn?rcdate=&q=%BF%C0%B4%C3%C0%C7+%C1%F5%BD%C3%C0%CF%C1%A4&sm=title.basic&pd=1&stDateStart="                                              
# [25] "https://finance.naver.com/news/news_read.nhn?article_id=0004897771&office_id=018&mode=LSS3D&type=0&section_id=101&section_id2=258&section_id3=401"                          
# [26] "https://finance.naver.com/news/news_read.nhn?article_id=0004897770&office_id=018&mode=LSS3D&type=0&section_id=101&section_id2=258&section_id3=401"                          
# [27] "https://finance.naver.com/news/news_read.nhn?article_id=0003895149&office_id=011&mode=LSS3D&type=0&section_id=101&section_id2=258&section_id3=401"                          
# [28] "https://finance.naver.com/news/news_read.nhn?article_id=0000949938&office_id=215&mode=LSS3D&type=0&section_id=101&section_id2=258&section_id3=401"                          
# [29] "https://finance.naver.com/news/news_read.nhn?article_id=0004897769&office_id=018&mode=LSS3D&type=0&section_id=101&section_id2=258&section_id3=401"                          
# [30] "https://finance.naver.com/news/news_read.nhn?article_id=0004897767&office_id=018&mode=LSS3D&type=0&section_id=101&section_id2=258&section_id3=401"                          
# [31] "https://finance.naver.com/news/news_read.nhn?article_id=0003895148&office_id=011&mode=LSS3D&type=0&section_id=101&section_id2=258&section_id3=401"                          
# [32] "https://finance.naver.com/news/news_read.nhn?article_id=0004897765&office_id=018&mode=LSS3D&type=0&section_id=101&section_id2=258&section_id3=401"                          
# [33] "https://finance.naver.com/news/news_read.nhn?article_id=0003895146&office_id=011&mode=LSS3D&type=0&section_id=101&section_id2=258&section_id3=401"                          
# [34] "https://finance.naver.com/news/news_read.nhn?article_id=0004897763&office_id=018&mode=LSS3D&type=0&section_id=101&section_id2=258&section_id3=401"                          
# [35] "https://finance.naver.com/news/news_list.nhn?mode=LSS2D&section_id=101&section_id2=258/javascript:view_newsflash_move('prev');"                                                                   
# [36] "https://finance.naver.com/news/news_list.nhn?mode=LSS2D&section_id=101&section_id2=258/javascript:view_newsflash_move('next');"
# [37] "https://finance.naver.com/news/news_read.nhn?article_id=0004897771&office_id=018&mode=LSS2D&type=0&section_id=101&section_id2=258&section_id3=&date=20210409&page=1"
# [38] "https://finance.naver.com/news/news_read.nhn?article_id=0004897770&office_id=018&mode=LSS2D&type=0&section_id=101&section_id2=258&section_id3=&date=20210409&page=1"
# [39] "https://finance.naver.com/news/news_read.nhn?article_id=0003895149&office_id=011&mode=LSS2D&type=0&section_id=101&section_id2=258&section_id3=&date=20210409&page=1"
# [40] "https://finance.naver.com/news/news_read.nhn?article_id=0004882120&office_id=277&mode=LSS2D&type=0&section_id=101&section_id2=258&section_id3=&date=20210409&page=1"
# [41] "https://finance.naver.com/news/news_read.nhn?article_id=0000949938&office_id=215&mode=LSS2D&type=0&section_id=101&section_id2=258&section_id3=&date=20210409&page=1"
# [42] "https://finance.naver.com/news/news_read.nhn?article_id=0004897769&office_id=018&mode=LSS2D&type=0&section_id=101&section_id2=258&section_id3=&date=20210409&page=1"
# [43] "https://finance.naver.com/news/news_read.nhn?article_id=0004897767&office_id=018&mode=LSS2D&type=0&section_id=101&section_id2=258&section_id3=&date=20210409&page=1"
# [44] "https://finance.naver.com/news/news_read.nhn?article_id=0003895148&office_id=011&mode=LSS2D&type=0&section_id=101&section_id2=258&section_id3=&date=20210409&page=1"
# [45] "https://finance.naver.com/news/news_read.nhn?article_id=0004897765&office_id=018&mode=LSS2D&type=0&section_id=101&section_id2=258&section_id3=&date=20210409&page=1"
# [46] "https://finance.naver.com/news/news_read.nhn?article_id=0003895146&office_id=011&mode=LSS2D&type=0&section_id=101&section_id2=258&section_id3=&date=20210409&page=1"
# [47] "https://finance.naver.com/news/news_read.nhn?article_id=0004897763&office_id=018&mode=LSS2D&type=0&section_id=101&section_id2=258&section_id3=&date=20210409&page=1"
# [48] "https://finance.naver.com/news/news_read.nhn?article_id=0010439801&office_id=003&mode=LSS2D&type=0&section_id=101&section_id2=258&section_id3=&date=20210409&page=1"
# [49] "https://finance.naver.com/news/news_read.nhn?article_id=0000680150&office_id=417&mode=LSS2D&type=0&section_id=101&section_id2=258&section_id3=&date=20210409&page=1"
# [50] "https://finance.naver.com/news/news_read.nhn?article_id=0004527819&office_id=015&mode=LSS2D&type=0&section_id=101&section_id2=258&section_id3=&date=20210409&page=1"
# [51] "https://finance.naver.com/news/news_read.nhn?article_id=0004527816&office_id=015&mode=LSS2D&type=0&section_id=101&section_id2=258&section_id3=&date=20210409&page=1"
# [52] "https://finance.naver.com/news/news_read.nhn?article_id=0010439794&office_id=003&mode=LSS2D&type=0&section_id=101&section_id2=258&section_id3=&date=20210409&page=1"
# [53] "https://finance.naver.com/news/news_read.nhn?article_id=0010439793&office_id=003&mode=LSS2D&type=0&section_id=101&section_id2=258&section_id3=&date=20210409&page=1"
# [54] "https://finance.naver.com/news/news_read.nhn?article_id=0010439791&office_id=003&mode=LSS2D&type=0&section_id=101&section_id2=258&section_id3=&date=20210409&page=1"
# [55] "https://finance.naver.com/news/news_read.nhn?article_id=0010439787&office_id=003&mode=LSS2D&type=0&section_id=101&section_id2=258&section_id3=&date=20210409&page=1"
# [56] "https://finance.naver.com/news/news_read.nhn?article_id=0010439786&office_id=003&mode=LSS2D&type=0&section_id=101&section_id2=258&section_id3=&date=20210409&page=1"
# [57] "https://finance.naver.com/news/news_list.nhn?mode=LSS2D&section_id=101&section_id2=258&page=1"                                                                                          
# [58] "https://finance.naver.com/news/news_list.nhn?mode=LSS2D&section_id=101&section_id2=258&page=2"                                                                                          
# [59] "https://finance.naver.com/news/news_list.nhn?mode=LSS2D&section_id=101&section_id2=258&page=3"                                                                                          
# [60] "https://finance.naver.com/news/news_list.nhn?mode=LSS2D&section_id=101&section_id2=258&page=4"                                                                                          
# [61] "https://finance.naver.com/news/news_list.nhn?mode=LSS2D&section_id=101&section_id2=258&page=5"                                                                                          
# [62] "https://finance.naver.com/news/news_list.nhn?mode=LSS2D&section_id=101&section_id2=258&page=6"                                                                                          
# [63] "https://finance.naver.com/news/news_list.nhn?mode=LSS2D&section_id=101&section_id2=258&page=7"                                                                                          
# [64] "https://finance.naver.com/news/news_list.nhn?mode=LSS2D&section_id=101&section_id2=258&page=8"                                                                                          
# [65] "https://finance.naver.com/news/news_list.nhn?mode=LSS2D&section_id=101&section_id2=258&page=9"                                                                                          
# [66] "https://finance.naver.com/news/news_list.nhn?mode=LSS2D&section_id=101&section_id2=258&page=10"                                                                                         
# [67] "https://finance.naver.com/news/news_list.nhn?mode=LSS2D&section_id=101&section_id2=258&page=11"                                                                                         
# [68] "https://finance.naver.com/news/news_list.nhn?mode=LSS2D&section_id=101&section_id2=258&page=31"                                                                                         
# [69] "https://finance.naver.com/news/news_list.nhn?mode=LSS2D&section_id=101&section_id2=258&date=20210408"                                                                                   
# [70] "https://finance.naver.com/news/news_list.nhn?mode=LSS2D&section_id=101&section_id2=258&date=20210407"                                                                                   
# [71] "https://finance.naver.com/news/news_list.nhn?mode=LSS2D&section_id=101&section_id2=258&date=20210406"                                                                                   
# [72] "https://finance.naver.com/news/news_list.nhn?mode=LSS2D&section_id=101&section_id2=258&date=20210405"                                                                                   
# [73] "https://finance.naver.com/news/news_read.nhn?article_id=0003606973&office_id=023&mode=RANK&typ=0"                                                                                       
# [74] "https://finance.naver.com/news/news_read.nhn?article_id=0004570324&office_id=008&mode=RANK&typ=0"                                                                                       
# [75] "https://finance.naver.com/news/news_read.nhn?article_id=0001429037&office_id=005&mode=RANK&typ=0"                                                                                       
# [76] "https://finance.naver.com/news/news_read.nhn?article_id=0004570379&office_id=008&mode=RANK&typ=0"                                                                                       
# [77] "https://finance.naver.com/news/news_read.nhn?article_id=0005278879&office_id=421&mode=RANK&typ=0"                                                                                       
# [78] "https://finance.naver.com/news/news_read.nhn?article_id=0004777118&office_id=009&mode=RANK&typ=0"                                                                                       
# [79] "https://finance.naver.com/news/news_read.nhn?article_id=0003894898&office_id=011&mode=RANK&typ=0"                                                                                       
# [80] "https://finance.naver.com/news/news_read.nhn?article_id=0001819347&office_id=016&mode=RANK&typ=0"                                                                                       
# [81] "https://finance.naver.com/sise/lastsearch2.nhn"                                                                                                                                                     
# [82] "https://finance.naver.com/item/main.nhn?code=005930"                                                                                                                                                
# [83] "https://finance.naver.com/item/main.nhn?code=000890"                                                                                                                                                
# [84] "https://finance.naver.com/item/main.nhn?code=035720"                                                                                                                                                
# [85] "https://finance.naver.com/item/main.nhn?code=011200"                                                                                                                                                
# [86] "https://finance.naver.com/item/main.nhn?code=302440"                                                                                                                                                
# [87] "https://finance.naver.com/rules.nhn"                                                                                                                                                                
# [88] "https://finance.naver.com/news/news_list.nhn?mode=LSS2D&section_id=101&section_id2=258/javascript:;"                                                                                                

# $ExternalLinks
# [1] "https://www.naver.com/"                                                                                       
# [2] "http://www.seibro.or.kr/websquare/control.jsp?w2xPath=/IPORTAL/user/company/BIP_CNTS01020V.xml&menuNo=273"
# [3] "https://www.naver.com/rules/service.html"                                                                     
# [4] "https://www.naver.com/rules/privacy.html"                                                                     
# [5] "https://www.naver.com/rules/disclaimer.html"                                                                  
# [6] "https://help.naver.com/support/alias/contents2/finance/finance_1.naver"                                       
# [7] "https://www.navercorp.com/"

Xpath

//*[@id=”contentarea_left”]/ul/li[1]/dl/dd[1]/a ==> //*[@id=”contentarea_left”]/ul/li/dl/dd/a

playing it safely with a Local HTTP server

terms of service

robots.txt

웹크롤링하려는웹사이트의메인페이지에서사전에‘robots.txt’를
확인해야하며,특히수집한데이터를영업에사용할목적이라면반드시법률
검토를진행하시기바랍니다

https://warm-uk.tistory.com/39 [데이터 크롤링] OPEN API 네이버 검색 데이터 crawling 하기_이해하기

Crawl Gently

Rcrawler( url, no_cores=1 , RequestsDelay=2, MaxDepth = 1)

process갯수조절 : no_cores=1
천천히 : RequestsDelay = 2
Limit depth : MaxDepth = 1

https://rdrr.io/github/salimk/Rcrawler/f/README.md

view(INDEX)

Rcrawler::Crawling

https://www.pluralsight.com/guides/web-crawling-in-r

R을 이용한 퀀트 투자 포트폴리오 만들기 – Chapter 4 크롤링 이해하기

Xpath

terms of service

robots.txt

Crawl Gently

onesixx

quant

crawling

quant Python Basic

scraping

quant

Crawling :: example (clien.net)

Rcrawler::Crawling

Xpath

terms of service

robots.txt

Crawl Gently

onesixx

Related Posts