nginx转发豆瓣内容

事情是这样的,我每周末都会到处寻找有意思的活动参加,豆瓣活动是我的一个重要寻找来源之一。但是那个链接每次都记不住,正好搭建了博客,想直接挂在自己页面上,就方便多了。说干就干,一个iframe 就搞定的事情,分分钟解决。

可能是因为我平时都是在手机上看,豆瓣活动页面是有专门的手机版的,总感觉这个很难看,想要手机版。一开始以为只要改了`iframe`的宽度,豆瓣就会直接识别成手机版,然而事实并不是这样。查看请求发现,豆瓣是服务端渲染的,也就是说,页面到浏览器之前,就已经判定为pc 版或者手机版了。看来是通过请求头里的user-agent 字段来判断的。那就修改请求的ua 呗。查了一圈,iframe 无法修改ua……

这就比较麻烦了,得通过服务端转发一下。其实这个需求挺通用的,一直想好好写一个转发服务来着。不过最近还是比较忙,先不搞了,简单用nginx 配置一下转发吧。

location /douban/ {
        proxy_pass https://beijing.douban.com/;
        proxy_set_header Referer https://beijing.douban.com;
        proxy_set_header User-Agent "Mozilla/5.0 (Linux; Android 6.0; NEM-AL10 Build/HONORNEM-AL10; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/57.0.2987.132 MQQBrowser/6.2 TBS/043906 Mobile Safari/537.36 MicroMessenger/6.6.1.1220(0x26060133) NetType/WIFI Language/zh_CN";
    }

就这样5行代码搞定。跑起来一看,还可以,但是一点击链接,发现不是那么回事,链接里地址还是 https://beijing.douban.com/,跳转一次之后,页面又变成pc 版了,蛋疼。要是能把html 里的链接也替换掉,不就搞定了吗?

查了一下资料,几行代码搞定。不过这几行代码可是花了我好多时间。

location /douban/beijing/ {
        proxy_pass https://beijing.douban.com/; # 反向代理
        sub_filter_once off;                    # 全局替换
        sub_filter 'beijing.douban.com' 'www.xxx.cn/douban/beijing'; #字符串替换
        proxy_set_header Accept-Encoding ""; # 禁用gzip才能替换成功
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header Referer https://beijing.douban.com;
        proxy_set_header User-Agent "Mozilla/5.0 (Linux; Android 6.0; NEM-AL10 Build/HONORNEM-AL10; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/57.0.2987.132 MQQBrowser/6.2 TBS/043906 Mobile Safari/537.36 MicroMessenger/6.6.1.1220(0x26060133) NetType/WIFI Language/zh_CN";
    }

点击详情页的时候,坑爹的事情又发生了,详情页的链接居然是www.douban.com,当然是还需要再加点东西了。为了不多写代码,想到了用变量。

    location ~* /douban/(.+)/(.*) {
        proxy_pass https://$1.douban.com/$2?$args;
		resolver 8.8.8.8;//增加DNS解析地扯
        sub_filter 'beijing.douban.com' 'www.xxx.cn/douban/beijing';
        sub_filter 'www.douban.com' 'www.xxx.cn/douban/www';
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header Referer https://beijing.douban.com;
        proxy_set_header User-Agent "Mozilla/5.0 (Linux; Android 6.0; NEM-AL10 Build/HONORNEM-AL10; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/57.0.2987.132 MQQBrowser/6.2 TBS/043906 Mobile Safari/537.36 MicroMessenger/6.6.1.1220(0x26060133) NetType/WIFI Language/zh_CN";
    }

然而这一次我死活没成功,搞了一晚上,一直报一个错误:

1 * beijing could not be resolved (3: Host not found)

查资料说要增加 resolver 解析地址,我加了,但是还是没成功.

➜  ~ dig @8.8.8.8 beijing.douban.com

; <<>> DiG 9.10.3-P4-Ubuntu <<>> @8.8.8.8 beijing.douban.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 40740
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;beijing.douban.com.            IN      A

;; ANSWER SECTION:
beijing.douban.com.     299     IN      A       154.8.131.172
beijing.douban.com.     299     IN      A       154.8.131.165
beijing.douban.com.     299     IN      A       154.8.131.171

;; Query time: 76 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Fri Jan 17 02:02:41 CST 2020
;; MSG SIZE  rcvd: 95

我怀疑是nginx 把域名解析成beijing 了。

➜  ~ dig @8.8.8.8 beijing           

; <<>> DiG 9.10.3-P4-Ubuntu <<>> @8.8.8.8 beijing
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 23333
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;beijing.                       IN      A

;; AUTHORITY SECTION:
.                       86398   IN      SOA     a.root-servers.net. nstld.verisign-grs.com. 2020011601 1800 900 604800 86400

;; Query time: 39 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Fri Jan 17 02:04:09 CST 2020
;; MSG SIZE  rcvd: 111

重启了几十次调试,还是没成功。实在是太晚了,明天还得上班,先用笨办法顶上吧,有谁要是有解决方案,麻烦指教一下~

location /douban/beijing/ {
        proxy_pass https://beijing.douban.com/; # 反向代理
        sub_filter_once off;              # 全局替换
        sub_filter 'beijing.douban.com' 'www.redream.cn/douban/beijing'; #字符串替换
        sub_filter 'www.douban.com' 'www.redream.cn/douban/www';
        proxy_set_header Accept-Encoding ""; # 禁用gzip才能替换成功
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header Referer https://beijing.douban.com;
        proxy_set_header User-Agent "Mozilla/5.0 (Linux; Android 6.0; NEM-AL10 Build/HONORNEM-AL10; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/57.0.2987.132 MQQBrowser/6.2 TBS/043906 Mobile Safari/537.36 MicroMessenger/6.6.1.1220(0x26060133) NetType/WIFI Language/zh_CN";
    }

    location /douban/www/ {
        proxy_pass https://www.douban.com/;
        sub_filter_once off;
        sub_filter 'beijing.douban.com' 'www.redream.cn/douban/beijing';
        sub_filter 'www.douban.com' 'www.redream.cn/douban/www';
        proxy_set_header Accept-Encoding "";
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header Referer https://beijing.douban.com;
        proxy_set_header User-Agent "Mozilla/5.0 (Linux; Android 6.0; NEM-AL10 Build/HONORNEM-AL10; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/57.0.2987.132 MQQBrowser/6.2 TBS/043906 Mobile Safari/537.36 MicroMessenger/6.6.1.1220(0x26060133) NetType/WIFI Language/zh_CN";
    }

更新一下一些新的方法

  • 使用include
server {
    location /first/location/ {
        include shared.conf;
    }
    location /second/location/ {
        include shared.conf;
    }
}
  • 使用@引用
error_page 418 = @common_location;
location /first/location/ {
    return 418;
}
location /second/location/ {
    return 418;
}
location @common_location {
    # The common configuration...
}
  • 使用嵌套location
location /specialpages/ {
    # some config
    location /specialpages/static/ {
        try_files $uri $uri/ =404;
    }
    location /specialpages/dynamic/ {
        proxy_pass http://127.0.0.1;
    }
}

放上效果地址:北京周末去哪玩

参考文档:

* https://juejin.im/post/5bd7a6046fb9a05d2c43f8c7#heading-1

* https://moonbingbing.gitbooks.io/openresty-best-practices/content/ngx/nginx_brief.html

Leave a Comment

电子邮件地址不会被公开。 必填项已用*标注