Skip to content
프로젝트

lucasxlu/LagouJob has 261 GitHub stars and 124 forks. GitHot measured 0 stars in 24 hours and 0 in 7 days. Last checked 2026-11-20 07:18 UTC. Last push 2019-04-19.

이 수치를 측정하는 방법

측정된 스타 증가

in 24h
0
in 7 days
0
in 15 days
기록 수집 중
in 1 month
기록 수집 중

서로 겹치는 기간의 순증가량이며 일별 합계가 아닙니다. 측정값이 없으면 비워 둡니다. 스타는 관심도를 나타내며 품질을 보장하지 않습니다.

소개

Data Analysis & Mining for lagou.com

머신러닝

README.md

Original

Data Analysis of Lagou Job

Lagou

Introduction

This repository holds the code for job data analysis of Lagou. The main functions included are listed as follows:

  1. Crawling job data from Lagou, and get the latest information of jobs about Internet.
  2. Proxies are collected from XiCiDaiLi.
  3. Data analysis and visualization.
  4. Crawling job details info and generate word cloud as Job Impression.
  5. In order to train a NLP task with machine learning, the data of interviewee's comments will be stored in mongodb

Prerequisites

  1. Install 3rd party libraries

    sudo pip3 install -r requirements.txt

  2. Install mongodb and start mongodb service [optional]

    sudo service mongod start

How to Use

  1. clone this project from github.
  2. Lagou's anti-spider strategy has been upgrade frequently recently. I suggest you run proxy_crawler.py to get IP proxies and execute the code with PhantomJS.
  3. run m_lagou_spider.py to crawl job data, it will generate a collection of Excel files in ./data directory.
  4. run hot_words_generator.py to cut sentences, it will return TOP-30 hot words and wordcloud figure.

Analysis Results

Image1
Image2
Image3
Image4
Image5
Image6
Image7

Report

  • For technical details, please refer to my answer at Zhihu.
  • The PDF report can be downloaded from here.

Change Log

  • [V2.0] - 2019.04. Upgraded to PhantomJS and IP proxies.
  • [V1.2] - 2017.05. Rewrite WordCloud visualization module.
  • [V1.0] - 2017.04. Upgraded to mobile Lagou.
  • [V0.8] - 2016.05. Finish Lagou PC web spider.

LICENSE

Apache-2.0